Method and system for personalized driving lane planning in autonomous driving vehicles

ABSTRACT

The present teaching relates to method, system, medium, and implementation of lane planning in an autonomous vehicle. Sensor data are received that capture ground images of a road the autonomous vehicle is on. Based on the sensor data, a current lane of the road that autonomous vehicle is currently occupying is detected. Information indicating presence of a passenger in the vehicle is obtained and used to retrieve a personalized lane control model related to the passenger. Lane control for the autonomous vehicle is planned based on the detected current lane and self-aware capability parameters in accordance with the personalized lane control model. The self-aware capability parameters are used to predict operational capability of the autonomous vehicle with respect to a current location of the autonomous vehicle. The personalized lane control model is generated based on recorded human driving data.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of U.S. patent applicationSer. No. 15/845,294 filed Dec. 18, 2017 and is related to U.S. patentapplication Ser. No. 15/845,173 filed Dec. 18, 2017, U.S. patentapplication Ser. No. 15/845,337 filed on Dec. 18, 2017 and U.S. patentapplication Ser. No. 15/845,423 filed on December 18, all of which areincorporated herein by reference in their entireties.

BACKGROUND 1. Technical Field

The present teaching generally relates to autonomous driving. Morespecifically, the present teaching relates to planning and control inautonomous driving.

2. Technical Background

With recent technological advancement in artificial intelligence (AI),there is a surge in applying AI in different application fields. Thisincludes the field of autonomous driving, in which planning and controlare essential. This is shown in FIG. 1 (Prior Art), in which anautonomous driving module 110 includes a planning module 120 and avehicle control module 130. Planning may include, as shown in FIG. 2,route planning, motion planning, or behavior planning. Route planningrefers to the effort to plan a route from a source to a destinationbased on certain considerations.

Motion planning may generally refer to the effort of planning themovement of a vehicle to achieve certain effect. For example, themovement of the vehicle may be planned in a way that complies with thetraffic regulations or safety. Motion planning is then to determine whatmovement the vehicle needs to make to achieve that. Behavior planninggenerally refers to the effort to plan how the vehicle should behave indifferent situations, e.g., the vehicle behavior while crossing anintersection, the vehicle behavior in staying within or following alane, or the vehicle behavior in making a turn. For instance, in termsof overtaking a slow moving front vehicle, certain vehicle behavior maybe planned. Behavior planning and motion planning may be related. Forexample, the planned vehicle behavior may need to be translated intomotion in order to implement the behavior.

Vehicle control 130 as shown in FIG. 1 may involve various aspects ofcontrol. This is illustrated in FIG. 3, which shows that vehicle controlmay involve, e.g., roadway specific control, motion specific control,mass specific control, geometry specific control, aerodynamic specificcontrol, and tire specific control.

Surrounding information 100 in FIG. 1 may be used for vehicle planning.Traditionally, surrounding information 100 includes, e.g., currentlocation of the vehicle, intended destination, and/or trafficinformation. Utilizing such surrounding information, the conventionalplanning module 120 may devise, e.g., a plan for a route from thecurrent location to the destination. Known criteria used in routeplanning may include, e.g., shortest distance, shortest time, use ofhighways, use of local roads, traffic, etc. Such criteria may be appliedbased on known information such as the distance of each road segment,known traffic patterns associated with roads, etc.

The planning module 120 may also perform motion planning, which istraditionally based on, e.g., rapidly exploring random trees (RRT) forstate space or Markov Decision Process (MDP) for environmental modeling.The planning module 120 may generate, based on the planned route/motion,planning data to be fed to the vehicle control module 130 so that thevehicle control module 130 can proceed to control the vehicle in a wayas planned. To make the vehicle to move to carry out the plan, thevehicle control module 130 may then generate control signals 140 whichmay be sent to different parts of the vehicle to implement the plannedvehicle movement. Vehicle control is traditionally exercised based ongeneric vehicle kinematic models and/or different types of feedbackcontrollers.

Each human driver generally operates or controls a vehicle differentlywith diverse preferences. Human drivers also operate vehicles adaptivelybased on real time situations, which may arise out of the presentconditions of the vehicle itself, the extrinsic environment conditionsthat serve to limit the ability of the vehicle to operate, and/or thereaction or response to the current vehicle movement from passengers inthe vehicle. For example, with children in the vehicle, a human drivermay elect, for safety, to avoid (route planning) a route that is curvyon a snowy day. A human driver may drive in different manners whendifferent passengers are riding in the vehicle to ensure comfort of thepassenger. Although a human driver generally controls a vehicle byfollowing a lane by staying roughly in the middle of the lane, thebehavior may change when faced with a right turn. In this case, the samehuman driver may curve to the right side of the lane when the vehicle isapproaching the point of the right turn. In addition, different humandrivers may curve to the right in different ways. Furthermore, lanechanging behavior may also differ with respect to different vehicles indifferent surrounding situations. The existing technologies do notaddress those issues, let alone providing solutions.

Therefore, there is a need to provide improved solutions for planningand control in autonomous driving.

SUMMARY

The teachings disclosed herein relate to methods, systems, andprogramming for online services. More particularly, the present teachingrelates to methods, systems, and programming for developing a virtualagent that can have a dialog with a user.

In one example, a method for lane planning in an autonomous vehicle isdisclosed. Sensor data are received that capture ground images of a roadthe autonomous vehicle is on. Based on the sensor data, a current laneof the road that autonomous vehicle is currently occupying is detected.Information indicating presence of a passenger in the vehicle isobtained and used to retrieve a personalized lane control model relatedto the passenger. Lane control for the autonomous vehicle is plannedbased on the detected current lane and self-aware capability parametersin accordance with the personalized lane control model. The self-awarecapability parameters are used to predict operational capability of theautonomous vehicle with respect to a current location of the autonomousvehicle. The personalized lane control model is generated based onrecorded human driving data.

In another example, a system for lane planning in an autonomous vehicleis disclosed. The system comprises a driving lane detector and a drivinglane planning unit. The driving lane detector is configured forreceiving sensor data capturing ground images of a road the autonomousvehicle is on and detecting, from the sensor data, a current lane of theroad the autonomous vehicle is currently occupying. The driving laneplanning unit is configured for obtaining information indicatingpresence of a passenger within the autonomous vehicle, retrieving apersonalized lane control model related to the passenger, and planninglane control for the autonomous vehicle based on the detected currentlane and self-aware capability parameters in accordance with thepersonalized lane control model. The self-aware capability parametersare used to predict the operational capability of the autonomous vehiclewith respect to a current location of the vehicle. The personalized lanecontrol model is generated based on recorded human driving data.

Other concepts relate to software for implementing the present teachingon developing a virtual agent. A software product, in accord with thisconcept, includes at least one machine-readable non-transitory mediumand information carried by the medium. The information carried by themedium may be executable program code data, parameters in associationwith the executable program code, and/or information related to a user,a request, content, or information related to a social group, etc.

In one example, machine readable non-transitory medium is disclosed,wherein the medium has information recorded thereon for lane control inan autonomous vehicle, where the information, when read by the machine,causes the machine to perform various steps. Sensor data are receivedthat capture ground images of a road the autonomous vehicle is on. Basedon the sensor data, a current lane of the road that autonomous vehicleis currently occupying is detected. Information indicating presence of apassenger in the vehicle is obtained and used to retrieve a personalizedlane control model related to the passenger. Lane control for theautonomous vehicle is planned based on the detected current lane andself-aware capability parameters in accordance with the personalizedlane control model. The self-aware capability parameters are used topredict operational capability of the autonomous vehicle with respect toa current location of the autonomous vehicle. The personalized lanecontrol model is generated based on recorded human driving data.

Additional novel features will be set forth in part in the descriptionwhich follows, and in part will become apparent to those skilled in theart upon examination of the following and the accompanying drawings ormay be learned by production or operation of the examples. The novelfeatures of the present teachings may be realized and attained bypractice or use of various aspects of the methodologies,instrumentalities and combinations set forth in the detailed examplesdiscussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The methods, systems and/or programming described herein are furtherdescribed in terms of exemplary embodiments. These exemplary embodimentsare described in detail with reference to the drawings. Theseembodiments are non-limiting exemplary embodiments, in which likereference numerals represent similar structures throughout the severalviews of the drawings, and wherein:

FIG. 1 (Prior Art) shows some essential modules in autonomous driving;

FIG. 2 illustrates exemplary types of planning in autonomous driving;

FIG. 3 illustrates commonly known types of vehicle control;

FIG. 4A depicts an autonomous driving vehicle with a planning module anda vehicle control module, according to an embodiment of the presentteaching;

FIG. 4B illustrates exemplary types of real time data, according to anembodiment of the present teaching;

FIG. 5 depicts an exemplary high level system diagram of a planningmodule, according to an embodiment of the present teaching;

FIG. 6A illustrates exemplary ways to realizing a safe-aware capabilitymodel, according to an embodiment of the present teaching;

FIG. 6B illustrates an exemplary construct of a self-aware capabilitymodel with parameters, according to an embodiment of the presentteaching;

FIG. 6C illustrates exemplary types of intrinsic vehicle capabilityparameters, according to an embodiment of the present teaching;

FIG. 6D illustrates exemplary types of extrinsic capability parameters,according to an embodiment of the present teaching;

FIG. 7 depicts an exemplary high level system diagram of a mechanism forgenerating self-aware capability parameters to be considered forplanning, according to an embodiment of the present teaching;

FIG. 8 depicts an exemplary high level system diagram of a self-awarecapability parameter generator, according to an embodiment of thepresent teaching;

FIG. 9 is a flowchart of an exemplary process for generating self-awarecapability parameters, according to an embodiment of the presentteaching;

FIG. 10 depicts an exemplary high level system diagram of a routeplanning module, according to an embodiment of the present teaching;

FIG. 11 is a flowchart of an exemplary process for route planning,according to an embodiment of the present teaching;

FIG. 12 depicts an exemplary high level system diagram of a global routeplanner, according to an embodiment of the present teaching;

FIG. 13 is a flowchart of an exemplary process for a global routeplanner, according to an embodiment of the present teaching;

FIG. 14A depicts an exemplary high level system diagram of a motionplanning module, according to an embodiment of the present teaching;

FIG. 14B illustrates exemplary types of passenger models, according toan embodiment of the present teaching;

FIG. 14C illustrates exemplary types of user reactions to be observedfor motion planning, according to an embodiment of the present teaching;

FIG. 15 depicts an exemplary high level system diagram of a passengerobservation analyzer, according to an embodiment of the presentteaching;

FIG. 16 is a flowchart of an exemplary process of a passengerobservation analyzer, according to an embodiment of the presentteaching;

FIG. 17 is a flowchart of an exemplary process for a motion planningmodule, according to an embodiment of the present teaching;

FIG. 18 depicts an exemplary high level system diagram of a modeltraining mechanism for generating different models for motion planning,according to an embodiment of the present teaching;

FIG. 19 illustrates different types of reactions to be observed andtheir roles in model training, according to an embodiment of the presentteaching;

FIG. 20A illustrates exemplary types of lane related planning, accordingto an embodiment of the present teaching;

FIG. 20B illustrates exemplary types of behavior related to lanefollowing, according to an embodiment of the present teaching;

FIG. 20C illustrates exemplary types of behavior related to lanechanging, according to an embodiment of the present teaching;

FIG. 21 depicts an exemplary high level system diagram of a laneplanning module, according to an embodiment of the present teaching;

FIG. 22 is a flowchart of an exemplary process for a lane planningmodule, according to an embodiment of the present teaching;

FIG. 23A illustrates the traditional approach to generate vehiclecontrol signals based on a vehicle kinematic model;

FIG. 23B depicts a high level system diagram of a vehicle control modulethat enables human-like vehicle control, according to an embodiment ofthe present teaching;

FIG. 23C depicts a high level system diagram of a vehicle control modulethat enables personalized human-like vehicle control, according to anembodiment of the present teaching;

FIG. 24 depicts an exemplary high level system diagram of a human-likevehicle control unit, according to an embodiment of the presentteaching;

FIG. 25 is a flow chart of an exemplary process of a human-like vehiclecontrol unit, according to an embodiment of the present teaching;

FIG. 26 depicts an exemplary high level system diagram of a human-likevehicle control model generator, according to an embodiment of thepresent teaching;

FIG. 27 is a flowchart of an exemplary process of a human-like vehiclecontrol model generator, according to an embodiment of the presentteaching;

FIG. 28 depicts an exemplary high level system diagram of a human-likevehicle control signal generator, according to an embodiment of thepresent teaching;

FIG. 29 is a flowchart of an exemplary process of a human-like vehiclecontrol signal generator, according to an embodiment of the presentteaching;

FIG. 30 depicts the architecture of a mobile device which can be used toimplement a specialized system incorporating the present teaching; and

FIG. 31 depicts the architecture of a computer which can be used toimplement a specialized system incorporating the present teaching.

DETAILED DESCRIPTION

In the following detailed description, numerous specific details are setforth by way of examples in order to provide a thorough understanding ofthe relevant teachings. However, it should be apparent to those skilledin the art that the present teachings may be practiced without suchdetails. In other instances, well known methods, procedures, components,and/or circuitry have been described at a relatively high-level, withoutdetail, in order to avoid unnecessarily obscuring aspects of the presentteachings.

The present disclosure generally relates to systems, methods, medium,and other implementations for planning and control of route/vehiclebehaviors in a self-capability aware, human-like, and personalizedmanner that is adaptive to real time situations. FIG. 4A shows anautonomous vehicle with a vehicle planning/control mechanism 410,according to an embodiment of the present teaching. The autonomousvehicle planning/control mechanism 410 includes a planning module 440and a vehicle control module 450. Both modules take various types ofinformation as input in order to achieve operations that areself-capability aware, human-like, personalized and adaptive to realtime situations. For example, as shown, both the planning module 440 andthe vehicle control module 450 receive historical manual driving data430 in order to learn human like ways to handle the vehicle in differentsituations. These modules also receive real time data 480 in order to beaware of the dynamic situations surround the vehicle in order to adaptthe operations accordingly. Furthermore, the planning module 440accesses a self-aware capability model 490 which characterizes whatlimits the operational ability of the vehicle given the situation thevehicle is currently in.

Real time data 480 may include various types of information useful orrelevant for planning and control of the vehicle. FIG. 4B illustratesexemplary types of real time data, according to an embodiment of thepresent invention. For example, exemplary real time data may includevehicle related data, time related data, passenger related data, weatherrelated data, . . . , and data related to the nearby roads. Vehiclerelated data may include, e.g., the motion state, position, orconditions of the vehicle at the time. The motion state of a vehicle mayinvolve, e.g., its current speed and driving direction. The real timeposition information may include, e.g., the current latitude, longitude,and altitude of the vehicle. The real time conditions of the vehicle mayinclude the functional state of the vehicle such as whether the vehicleis currently in a full or partial functional state or specificparameters under which different components of the vehicle areoperating, etc.

Real time data related to time may generally include current date, time,or month. Passenger related data may include various characteristicsrelated to the passenger of the vehicle such as passenger reaction cues,which may include visual, acoustic, or behavior cues observed from thepassenger, or conditions of the passenger such as mental state, physicalstate, or functional state of the passenger. The conditions of thepassenger may be inferred based on the cues observed from the passengerreaction cues. Weather related data may include the weather of thelocale where the vehicle is currently situated. The road related datamay include information about the physical condition of the nearbyroad(s), e.g., wetness, steepness, or curviness of the road, or thelocal traffic condition such as congestion along the road.

FIG. 5 depicts an exemplary high level system diagram of the planningmodule 440, according to an embodiment of the present teaching. In thisexemplary embodiment, the planning includes, but not limited to, routeplanning, motion planning, and the planning of lane related behavior,including lane following, lane changing, etc. Accordingly, in thisillustrated embodiment, the planning module 440 comprises a routeplanning module 550, a motion planning module 560, and a lane planningmodule 570. Each of the modules aims at operating in a self-capabilityaware, human-like, and personalized manner. Each of the modules 550,560, and 570 takes, in addition to the surrounding information 420,recorded human driving data 430, real time data 480, and the self-awarecapability model 490 as inputs and generates their respective outputs tobe used by the vehicle control module 450 to convert into the vehiclecontrol signals 470 to control the vehicle. For example, the routeplanning module 550 generates the planned route information 520 as itsoutput, the motion planning module 560 generates planned motion 530 asits output, and the lane planning module 570 generates planned lanecontrol information 540 as its output.

Each of the planning modules may be triggered via some triggeringsignal. For instance, the route planning module 550 may be activated viaa route planning trigger signal; the motion planning module 560 may beactivated upon receiving a motion planning trigger signal; while thelane planning module 570 may start to operate when a lane planningtrigger signal is received. Such a trigger signal may be manuallyprovided (by, e.g., a driver or a passenger) or automatically generatedbased on, e.g., certain configuration or certain event. A driver maymanually activate the route planning module 550 or any other planningmodule for the route/motion/lane planning, much like what people do tomanually start, e.g., cruise control in a car.

The planning activities may also be activated by a certain configurationor an event. For example, the vehicle may be configured to activateroute planning whenever the vehicle accepts an input indicating the nextdestination. This may be regardless what the current location of thevehicle is. In some embodiments, the planning modules may be alwaystriggered on whenever the vehicle is on and depending on the situation,they may become engaged in different planning activities as needed. Indifferent situations, they may also interact with each other in a mannercalled for by the situation. For example, the lane planning module 570may determine to change lane in certain circumstance. Such a plannedlane control is output by the lane planning module 570 and may be fed tothe motion planning module 560 so that a specific path trajectory(planned motion) appropriate for carrying out the planned lane changingmay be further planned by the motion planning module 560.

Output of a planning module may be fed into another within the planningmodule 440 for either further planning or for providing an input for thefuture planning of another. For example, the output of the routeplanning module 550 (planned route 520) may be fed to the motionplanning module 560 so that the route information may influence how thevehicle motion is planned. As discussed above, the output (planned lanecontrol 540) of the lane planning module 570 may be fed to the motionplanning module 560 so that the lane control behavior planned may berealized via planned motion control. Conversely, the output of themotion planning module 560 (the planned motion 530) may also be fed tothe lane planning module 570 to influence the planning of the lanecontrol behavior. For instance, in personalized motion planning, themotion planning module 560 may determine that the motion of the vehicleneeds to be gentle due to the observation that the passenger of thevehicle prefers smooth motion. Such a determination is part of themotion planning and may be to be sent to the lane planning module 570 sothat the lane control behavior of the vehicle may be carried out in away that ensures smooth motion, e.g., change lane as little as possible.

To ensure that the vehicle behavior is planned and controlled in aself-capability aware manner, the route planning module 550, the motionplanning module 560, and the lane planning module 570 also access theself-aware capability model 490 and use it to determine the planningstrategy in a manner that takes into account of what the vehicle isactually capable of in the current scenario. FIG. 6A illustratesexemplary ways that the self-aware capability model 490 is realized,according to an embodiment of the present teaching. As illustrated, theself-aware capability model 490 may be constructed as a probabilisticmodel, a parameterized model, or a descriptive model. Such a model maybe trained based on, e.g., learning. The model may include a variety ofparameters to be used to characterize factors that may influence or havean impact on the actual ability of the vehicle. The model may beimplemented as a probabilistic model with parameters being estimatedprobabilistically. The model may also be implemented as a parameterizedmodel with explicit model attributes applicable to different real worldconditions. The model 490 may also be provided as a descriptive modelwith enumerated conditions with values instantiated based on real timescenarios.

The self-aware capability model 490 in any situation may include variousparameters, each of which is associated with some factors that mayimpact the actual ability of the vehicle so that the vehicle planning(route, motion, or lane) has to consider. In the following disclosure,self-aware capability model and self-aware capability parameters will beused interchangeably. FIG. 6B illustrates an exemplary construct of theself-aware capability model or parameters 510, according to anembodiment of the present teaching. As illustrated, self-awarecapability parameters 510 may include intrinsic capability parametersand extrinsic capability parameters. Intrinsic vehicle capabilityparameters may refer to parameters associated with the vehicle itselfwhich may impact what the vehicle is capable of in operation and suchparameters may be determined based on either how the vehicle ismanufactured or how the vehicle is at the time. Extrinsic capabilityparameters may refer to the parameters or characteristics of thesurrounding that are extrinsic to the vehicle but nevertheless mayimpact the way the vehicle can be operated.

FIG. 6C illustrates exemplary types of intrinsic vehicle capabilityparameters, according to an embodiment of the present teaching. Asillustrated, intrinsic vehicle capability parameters may include, butnot limited to, characteristics of the vehicle in terms of, e.g., itsengine, its safety measures, and its tires, etc. For instance, in termsof its engine, the intrinsic capability parameters may specify themaximum speed the vehicle is capable of, the control that can beexercised on the engine, including cruise control or any restrictions onmanual control of the engine. In terms of safety measures, the intrinsiccapability parameters may include information on what sensors thevehicle is equipped with, specific parameters related to breaks, orinformation about the seats of the vehicle. For example, some vehiclesmay have seats that are backed by metal support (stronger) and some withonly plastic support. Some seats may have mechanism which allowsautomatic control to have vibrations and some may not. In terms of othercomponents of the vehicle, the intrinsic capability parameters may alsospecify the type of the tires of the vehicle (which may have a bearingon what operation can be done) and whether the vehicle currently hassnow tires installed or equipped with anti-skip measures. Such intrinsicvehicle capability parameters may be used to assess what types of routesand motions may be possible and which types of vehicle behaviors may beachievable. Thus, making such intrinsic capability parameters availableto the planning modules allows the planning modules to planappropriately without exceeding what the vehicle is actually capable of.

FIG. 6D illustrates exemplary types of extrinsic capability parameters,according to an embodiment of the present teaching. As discussed above,extrinsic capability parameters specify information that are external tothe vehicle but nevertheless may impact the ability of planning and suchextrinsic capability parameters are used to determine appropriateplanning given the conditions external to the vehicle. The ultimateoutput from the planning modules may be determined within the confine ofboth intrinsic and extrinsic capability parameters. As extrinsiccapability parameters may include parameters describing the conditionsor situations the vehicle is in or may encounter, they likely willimpact what should be planned. For example, the surrounding situationsrelated to the road(s), either close to the vehicle or even relativelyremote from the vehicle. The road condition related parameters mayindicate how crowded the road is (so that the driving speed cannot beplanned to be fast), whether there are speed limits on the road(regulated minimum and maximum speeds as well as the actual permissiblespeed due to traffic), whether there is any accident along the road, orthe surface condition of the road (so that the motion cannot be tooswift), or whether the surface of the road presents certain conditionsthat will impede the ability of the vehicle in its planning.

There are other conditions external to the vehicle that may affectvarious planning activities. This includes the light or atmosphererelated conditions as well as the surrounding of the vehicle. Forinstance, if the vehicle is positioned in such a way that there is a sunglare so that sensors may not work well, this will impact the planningdecisions. If the vehicle is in an area with a heavy fog condition, suchinformation is also important to the planning module. If there is a highlevel of precipitation, such information may also be taken into accountby the planning modules. The surrounding traffic may also be importantin terms of planning. For instance, extrinsic parameters may provideinformation related to nearby vehicles or objects so that the planningmodules may consider such information in their respective planning. Theextrinsic parameters may include information about such nearbyvehicles/objects, e.g., the nearby vehicle is a big truck or a bicycle,which may also impact how the planning decision is made. In addition,events occur along the road the vehicle is on may also impact theplanning. For instance, whether the vehicle is currently on a road thatis in a school zone or whether there is a construction going on alongthe road the vehicle is currently on may also be important informationto the planning modules for obvious reasons.

The extrinsic capability parameters may be acquired and updatedcontinuously in time to support the planning modules to adapt theirdecisions based on external situations in real time. In some situations,the extrinsic capability parameters may also be predicted. For example,if the vehicle is driving on a road to the west in the afternoon, it maybe predicted that there will be sun glare. Although such predictedextrinsic capability parameter may not be the real time information, itnevertheless will assist the planning module (e.g., the route planningmodule) to make an appropriate decision. For instance, if the intendeddestination for a vehicle is in the north west direction and there areroads available at the time to both the west and the north, knowing thatthere will be sun glare if heading to the west in late afternoon, theroute planning module 550 may accordingly decide to presently take theroad heading to the north first and later take a road to head to thewest after sun is down to avoid sun glare (safer). Such predictedextrinsic capability parameters may be determined based on otherinformation such as the current location of the vehicle and the intendeddestination of the vehicle.

With capability parameters (including both intrinsic and extrinsic), thevehicle becomes self-aware of both intrinsic and extrinsic capabilityrelated limitations, which may be crucial in terms of planning. FIG. 7depicts an exemplary high level system diagram for a mechanism 700 forgenerating self-aware capability parameters, according to an embodimentof the present teaching. In this illustrated embodiment, the mechanism700 comprises a locale context determining unit 730 and a self-awarecapability parameter generator 740. The locale context determining unit730 is to gather information locale to where the vehicle is and/or willbe (i.e., both where the vehicle is presently and where the vehicle willbe on its way to the destination) based on, e.g., information about thecurrent location of the vehicle and/or the destination the vehicle isheading to. The self-aware capability parameter generator 740 is togenerate both intrinsic and extrinsic capability parameters, e.g., on acontinuous basis, based on information related to the vehicle and thelocale context information determined based on, e.g., the current andfuture location of the vehicle.

To facilitate the self-aware capability parameter generator 740 togenerate extrinsic capability parameters, the locale context determiningunit 730 may retrieve information stored in a map configuration 750 anda road context configuration 760 based on the current location 720 andthe destination information 710. The locale context information relatedto the roads may include the surrounding or contextual information ofthe road the vehicle is currently on and/or the roads that the vehiclewill be on subsequently. For example, the map configuration 750 mayprovide information about the roads from the current location to theintended destination, while the road context configuration 760 mayprovide some known or static information about the characteristicsassociated with roads, such as altitude, steepness, curviness of eachroad, etc. Such gathered static information about the roads may then beused by the self-aware capability parameter generator 740.

The road conditions may change over time. For example, roads may becomeicy or slippery due to changes in weather conditions. Such dynamicallychanging context information about the roads may be acquired separatelyby, e.g., the self-aware capability parameter generator 740 on acontinuous basis and used in generating extrinsic capability parametersthat are reflective of the real time situations. As will be discussedbelow in reference to FIG. 8 about the self-aware capability parametergenerator 740, both the current location and the source-destinationinformation may also be sent to the self-aware capability parametergenerator 740 in order for it to gather real time information about roadconditions to determine the extrinsic capability parameters.

To generate intrinsic vehicle capability information, informationrelated to the vehicle may be accessed from a vehicle informationstorage 750. The vehicle information storage 750 may store vehicleparameters configured when the vehicle was manufactured such as whetherthe vehicle is equipped with cruise control or certain types of sensors.The storage 750 may also subsequently update information related to theparameters intrinsic to the vehicle. Such subsequent update may begenerated due to, e.g., vehicle maintenance or repair or even updateobserved in real time. In discussion below in reference to FIG. 8, theself-aware capability parameter generator 740 includes also themechanism to collect continuously any dynamic update of the vehiclerelated parameters consistent with the actual intrinsic capability ofthe vehicle.

FIG. 8 depicts an exemplary high level system diagram of the self-awarecapability parameter generator 740, according to an embodiment of thepresent teaching. In this illustrated embodiment, the self-awarecapability parameter generator 740 comprises a locale contextinformation processor 810, a situation parameter determiner 820, aself-aware capability parameter updater 830, and various updaters thatcontinuously and dynamically gather information of different aspectsrelated to decision making of the vehicle. Such dynamic informationupdaters include, e.g., vehicle capability parameter updater 860-a,weather sensitive parameter updater 860-b, traffic sensitive parameterupdater 860-c, orientation sensitive parameter updater 860-d, roadsensitive parameter updater 860-e, . . . , and time sensitive parameterupdater 860-f.

In some embodiments of the operation, upon receiving the locale contextinformation from the locale context determining unit 730, the localecontext information processor 810 processes the received informationand, e.g., extracts information related to the current route the vehicleis on and sends such information to the self-aware capability parameterupdater 830. Such information related to the current route may includesteepness or curviness of the route or other types of static informationsuch as the altitude and orientation of the route. The situationparameter determiner 820 receives the current location 720 and, e.g.,separates location and time information and sends the information to theself-aware capability parameter identifier 830 so that it may use thatinformation to identify capability parameters specific to the locationand the precise time.

With the information about the location of the vehicle and the currenttime, the self-aware capability parameter updater 830 may accessintrinsic capability models 840 and/or extrinsic capability models 850to retrieve capability related parameter values specific to the currentlocation and time. In some embodiments, the intrinsic capability models840 may be configured to specify types of parameters relevant to theintrinsic capabilities of the vehicle and the current values thereof.Similarly, the extrinsic capability models 850 may be configured tospecify types of parameters that have an impact on the ability of thevehicle to operate and their current values.

In operation, in order to keep the values of the parameters current, theintrinsic and extrinsic capability models (840 and 850) may regularlytrigger the updaters (860-a, . . . , 860-f) to gather real timeinformation and update the values of the corresponding parameters basedon such gathered real time information. For example, the intrinsiccapability models 840 may be configured to have a mechanism to activatethe vehicle capability parameter updater 860-a to gather updatedinformation related to the intrinsic capabilities of the vehicle. Such amechanism may specify different modes of triggering. For instance, itmay be on a regular schedule, e.g., daily or hourly. It may also specifyto be triggered by some external event such as a signal received from amaintenance shop or a sensor in the vehicle that senses that somefunctional state of a component in the vehicle has been changed. In thiscase, the vehicle capability parameter updater 860-a may accept realtime vehicle information from the sensor(s) and update the values/statesof the relevant capability parameter in the intrinsic capability modelsto reflect that real time status of the vehicle. For instance, if duringthe operation of the vehicle, the headlight or a break may becomenon-functional. Such information sensed in real time may be gathered bythe vehicle capability parameter updater 860-a and used to update theinformation stored in the intrinsic capability parameter storage 840.Such updated information relates to the vehicle may then be used by theself-aware capability parameters generator 740 to generate intrinsiccapability parameters.

Similarly, the extrinsic capability models 850 may be configured tospecify the update mechanism(s) for updating different types ofextrinsic capability parameters. The update mechanism may specifyregularly scheduled update or update to be triggered by some events.Different types of extrinsic capability parameters may be configured tobe updated based on different triggering mechanisms. For example, forweather related extrinsic capability parameters or extrinsic capabilityparameters that may be keyed on weather, e.g., visibility in thevicinity of the vehicle, the update may be made regularly, e.g., everyfew minutes. Similarly, traffic sensitive parameters, e.g., the actualpermissible speed which is usually direct result of the trafficcondition, may also be updated regularly. Different types of parameters,although all regularly updated, may have different update schedule,which may range from every few seconds to every few minutes or every fewhours.

On the other hand, some extrinsic capability related parameters may bemade upon the occurrence of some events. For instance, for anorientation sensitive parameter, e.g., whether sun glare exists, theupdate may be triggered when the vehicle is heading in certaindirections. If the direction of the vehicle is heading changes fromnorth to north-west at some afternoon time, this may trigger theorientation sensitive parameter updater 860-d to gather informationrelated to sun glare and update the situation with regard to sun glare.In some situations, the update may indicate that there is no sub glare,e.g., when it is a cloudy day. In some situations, the update mayindicate that there is sun glare. In either situation, such orientationsensitive information is then used to update the value of thecorresponding extrinsic capability parameter stored in the extrinsiccapability parameter storage 850. Similarly, update of time sensitiveparameters, such as visibility of the vehicle due to time of the day,may be triggered based on detected location, time zone of the location,and the specific time of the day at the moment. In some embodiments, theupdate of some of the capability parameters may also be triggered byevent related to the detected updates of other capability parametervalues. For example, the update of road sensitive parameters such asslippery road condition may be triggered when the update for the weathercondition indicates that it started to rain or snow.

In the illustrated embodiments, the vehicle capability parameter updater860-a receives the static vehicle information from storage 750 anddynamic vehicle information update from real time vehicle informationfeed which may be from multiple sources. Examples of such sourcesinclude dealers, vehicle maintenance places, sensors on the vehiclereporting the status change of components, or other sources. The weathersensitive parameter updater 860-b may receive both dynamic weatherupdate and the updates of other weather sensitive capability parameters,e.g., precipitation, visibility, fog, or any other parameters thatrelate to weather and have the potential to impact the operation of thevehicle. Weather related information may be from multiple data sourcesthat feed real time data.

The traffic sensitive parameter updater 860-c may receive both dynamictraffic reports and other information relating to the traffic that mayinfluence the operation of the vehicle. Examples include the extent ofthe traffic jam (which may be used to determine whether the route of thevehicle needs to be re-planned) or the time of the accident that hadcaused the traffic (to estimate how long the delay will continue inorder to determine whether to re-do the route planning). Traffic ortraffic related information may be received from one or more sources forreal time data feed. The orientation sensitive parameter updater 860-dmay be configured to gather information along the road in the directionof the vehicle. Such orientation sensitive information may include sunglare in certain directions (e.g., east or west) or any potentialsituations in the direction of the road the vehicle is on (e.g.,landslide situation ahead of the road). Similarly, the road sensitiveparameter updater 860-e may, once triggered, gather information aboutvarious roads or road conditions with respect to the location of thevehicle, from one or more real time information feed sources. Suchinformation may be related to the roads (e.g., open, close, detoured,school zone, etc.) or conditions thereof (e.g., slippery, icy, flooded,construction, etc.). The time sensitive parameter updater 860-f may beconfigured to collect from data source(s) real time data that depend ontime. For example, the visibility of the road may depend on the time ofday at the zone the vehicle is in.

The collected real time data may then be used to update the intrinsiccapability models 840 and/or the extrinsic capability models 850. Suchupdated data may be time stamped. The self-aware capability parameterupdater 830 may then access both the intrinsic and extrinsic capabilitymodels 840 and 850 to determine relevant capability parameters and theupdated values thereof. The retrieved intrinsic/extrinsic capabilityparameters may then be output so that they can be used by variousplanning modules as shown in FIG. 5. Specifically such generatedself-aware capability parameters 510 are used by the route planningmodule 550 for route planning, as will be discussed with reference toFIGS. 10-13. The self-aware capability parameters are also used by themotion planning module 560 for personalized motion planning, which willbe disclosed in detail with reference to FIGS. 14-19. The self-awarecapability parameters are also used by the lane planning module 570 forlane control, which will be detailed with reference to FIGS. 20-22.

FIG. 9 is a flowchart of an exemplary process of the self-awarecapability parameter generator 740, according to an embodiment of thepresent teaching. First, the locale context information is received at910 and location and time information are extracted at 920 and used at930 by different updaters to obtain information feeds from differentsources related to various aspects of intrinsic and extrinsiccapabilities. Such acquired information is then used by differentupdaters at 940 to update the intrinsic capability parameters 840 andextrinsic capability parameters 850. Based on the current location,time, and the received locale contextual information, the self-awarecapability parameter updater 830 then identifies various intrinsic andextrinsic capability parameters 510 relevant to the vehicle at thepresent time to update, at 940, the intrinsic/extrinsic capabilityparameters and generates, at 950, the updated capability parameters.Such updated intrinsic/extrinsic capability parameters 510 are thenoutput at 960.

Such dynamically gathered self-aware capability parameters are to beused in various vehicle behavior planning operations, including routeplanning, motion planning, and lane related vehicle behavior planning.For example, in human driving, choosing a route to a destination isoften done with consideration of factors captured by the self-awarecapability parameters. For example, a human driver may choose a route toa desired destination based on, e.g., what the vehicle is equipped withor capable of (intrinsic capability parameters). If the vehicle is insuch a condition that it cannot handle steep road well, then such roadneeds to be avoided. In addition, a human driver may also consider otherfactors such as weather of the day, conditions of the roads considered,events known or scheduled at certain time of day (extrinsic capabilityparameters). For instance, it one road points to the west and the sunwill be setting at that time, perhaps there will be too much glare sothat it is better to take another alternative road. For both safety andreliability, autonomous vehicles ought to also consider such intrinsicand extrinsic capabilities with respect to route choices during routeplanning.

Traditional approaches to route planning often adopt some cost functionso that the cost of a route selected is minimized. For instance,conventional route planning considers, e.g., optimization of distancetraveled, minimization of time required to arrive the destination, orminimize the fuel used to get to the destination. In some instances,conventional approaches may also consider traffic conditions inoptimizing the cost, e.g., high traffic route may decrease the speedleading to increased time and fuel to get to the destination. Suchoptimization functions often assume that all vehicles can handle allroutes in the same manner and all routes can be handled equally well.Such assumptions are often not true so that when autonomous vehiclesapply such planning schemes, they often find unable to proceed or evenbecome unsafe in some situations. The present teaching aims to achievesafe, realistic, and reliable route planning that is adaptive to thechanging intrinsic and extrinsic capability related parameters.

As shown in FIG. 5, the self-aware capability parameters 510 areconsidered by the planning module 450 in achieving different planningtasks, including the route planning module 550, the motion planningmodule 560, and the lane planning module 570. Below, in reference toFIGS. 10-13, details about the route planning module 550 are provided.FIG. 10 depicts an exemplary high level system diagram of the routeplanning module 550, according to an embodiment of the present teaching.The purpose of the route planning module 550 is to plan a route based ondesired destination in a self-aware manner in terms of both intrinsicand extrinsic capabilities. In contrast, conventional route planningtechnologies consider mainly criteria such as shortest distance,shortest time, most use of highways/local ways, etc. without taking intoaccount the dynamic intrinsic capability parameters and the real timeextrinsic capability parameters.

In this illustrated embodiment, the route planning module 550 comprisesa route selection preference determiner 1030 and a global route planner1020. The route selection preference determiner 1030 is to determine thepreferences to be considered in selecting a route. The global routeplanner 1020 is to select an appropriate route based on a variety ofinformation, including the self-aware capability parameters 150. In someembodiments, the route planning activities may be triggered based on theroute planning trigger signal as shown. Upon being activated, the globalroute planner 1020 may gather various types of dynamic informationrelevant to the present route planning operation. For example, theglobal route planner 1020 may rely on information related to thesource/current location and the destination desired. The planning isperformed with respect to the source/current location and thedestination. The destination information may be determined in differentways. For example, it may optionally be received from a driver/passengervia an interface unit 1010.

The global route planner 1020 may also take real time data 480 as inputand plans a route accordingly. As discussed with respect to FIG. 4B,real-time data include information related to real time vehicle relatedinformation (position), information about passenger observed in thevehicle, . . . , and road characteristics. Such real-time data providesurrounding information needed for the route planning. The global routeplanner 1020 also receives the self-aware capability parameters 510 thatinform the planner what is possible given the dynamic intrinsic andextrinsic situations at the time of the planning. For instance,intrinsic capability parameters may indicate that the vehicle iscurrently not able to run fast due to some mechanical problems so thatthe global route planner 1020 may take that into account to, e.g., plana route that involves mainly local roads and may possibly pass some carrepair shops. Similarly, extrinsic capability parameters may indicatethat in the north direction of the current location of the vehicle, thesun glare is quite severe so that the global route planner may base thatinformation to avoid a nearby route that is in the north directionbefore the sun is set. The real-time data 480 and the self-awarecapability parameters 510 provide information to the global routeplanner 1020 to enable it to plan a route that is appropriate given,e.g., the present time, the present location of the vehicle, the presentweather, the present passenger's situation, and present road conditions.

The global route planner 1020 may also consider preference(s) to beapplied in route planning. Such preferences may be specified by adriver/passenger via the user interface unit 1010 (which may beforwarded to the global route planner 1020) or may be obtained via othermeans (see disclosure below with reference to FIG. 12). In consideringthe preferences to be applied, information stored in route selectionpreference configuration 1050 may also be accessed and considered. Suchroute selection preference configuration may specify some generalpreference in route selection in different scenarios, e.g., avoidsteep/curvy roads in raining/snow scenarios, avoid small streets atnight, avoid roads with very few gas stations, etc. The global routeplanner 1020 may forward relevant information received from real-timedata 480 and self-aware capability parameters 510 to the route selectionpreference determiner 1030, which is in turn used by the route selectiondeterminer 1030 to retrieve certain route selection preferenceconfiguration from 1050. For example, if it is currently snowing (fromreal-time data 480) and the vehicle has no snow tire (from the intrinsiccapability parameter 510), such dynamic information may be forwardedfrom the global route planner 1020 to the route selection preferencedeterminer 1030 so that selection preference configuration related tosuch dynamic scenarios may be retrieved from the route selectionpreference configuration 1050 (e.g., avoid steep/curvy road) and sendsback to the global route planner 1020 so that it can be relied on inselecting an appropriate route.

To determine an appropriate route, in addition to knowing the selectionpreferences, the global route planner 1020 may also need to knowadditional information about the roads, such as what routes availablefrom the current location of the vehicle to the intended destination. Inaddition, for each route available, the map/road configuration 1060 mayalso store characteristic information about each of the availableroads/routes. Such characteristic information of the roads/routes mayinclude, but not limited to, geometric characteristics such as nature ofthe roads/routes (highway or not), dimension of the roads/routes,steepness/curviness, condition of the roads/routes, etc. Duringplanning, the global route planner 1020 may first determine theroads/routes available between the current location of the vehicle tothe desired destination. To select an appropriate route to thedestination, for such available roads/routes, their characteristicinformation may also be accessed by the global route planner 1020 sothat selections may be made based on such characteristic information.

With the information about the available roads/routes available as wellas the characteristic information about those available roads/routes,the global route planner 1020 may then select an appropriate route tothe destination by matching the route selection preferences, determinedby the route selection preference determiner 1030, with thecharacteristic information of the available roads/routes. Details aboutthe global route planner 1020 are provided with reference to FIGS.12-13.

As discussed previously, the global route planner 1020 selects a plannedroute based on dynamic information from different sources, includingreal-time data 480 and self-aware capability parameters 510. In additionto that, as the vehicle may be on move or the destination may changeover time, the current location of the vehicle and the destination mayalso change in time, just like the real-time data 480 and the self-awarecapability parameters 510. When such information changes, it may impactthe global route planned. For example, when the current locationchanges, the real-time data associated with the current location mayalso change, e.g., from good weather associated with the previous spotto the raining condition associated with the current location. This mayin turn lead to a change in terms of route selection preferences andultimately, the route selected. Thus, the global route planner 1020 mayinteract with the route selection preference determiner 1030 in abi-directional manner and dynamic manner. Whenever there is a changethat may warrant a re-determination of route selection preferences, theglobal route planner 1020 may then activate the route selectionpreference determiner 1030 to modify or re-generate the preferences tobe used by the global route planner 1020 to determine an appropriateroute given the situation.

FIG. 11 is a flowchart of an exemplary process for the route planningmodule 550, according to an embodiment of the present teaching.Information about the destination of the vehicle and optionally aboutthe preferences is received at 1110. Real-time data 480 as well asself-aware capability parameters 510 are received by the global routeplanner 1020 at 1120 and various information related to the currentscenario or situation of the vehicle may then be identified, at 1130,from the received real-time data and the self-aware capabilityparameters. Based on the relevant information related to the currentscenario, preferences specific to the current scenario are determined,at 1140. To plan a route, the global route planner 1020 accesses, at1150, information about available roads/routes with respect to thecurrent location and the desired destination as well as thecharacteristic information of such available roads/routes. At 1160,based on the specific preferences determined based on the currentscenario as well as the roads/routes information, the global routeplanner 1020 selects a route appropriate for the current situation.

FIG. 12 depicts an exemplary high level system diagram of the globalroute planner 1020, according to an embodiment of the present teaching.In this illustrated embodiment, the global route planner 1020 comprisesa self-aware capability parameter analyzer 1205, an intrinsic capabilitybased filter generator 1210, and a route selection engine 1230.Optionally, the global route planner 1020 also comprises a destinationupdater 1225 for dynamically determine and update the currentdestination. In the illustrated embodiment, the global route planner1020 also optionally include a mechanism for personalizing preferencesof a driver/passenger so that the route in selecting a route. The routeselection preference determiner 1030 is to determine the preferencesrelated to selecting a route based on the specific situation the vehicleis currently in, which differs from obtaining personalized preferencesdirected to a specific driver/passenger.

Illustratively, the optional mechanism to determine personalizedpreferences comprises a passenger driving data analyzer 1245, apreference personalization module 1250, and a passenger preferencedeterminer 1240. In operation, the passenger driving data analyzer 1245receives recorded human driving data 430 as input and analyzes or learnsfrom such data to understand route preferences relevant to specificdrivers/passengers. For example, from the recorded human driving data430, it may be learned that a specific driver prefers to drive on localroads instead of highway or historically chose to use highways at nightseven though it involved much longer distance. It may also learnpreferences of all drivers associated with the vehicle. For instance,multiple people (husband, wife, and a child of a household) may beassociated with the vehicle, i.e., anyone of these people may operatethe vehicle. The passenger driving data analyzer 1245 may learn from therecorded human driving data 430 various types of information associatedwith the driving behavior of such drivers, which may enable thepreference personalization module 1250, upon receiving such drivingbehavior information, to establish personal preferences of each of suchindividuals.

Upon receiving the information related to each individual driver fromthe passenger driving data analyzer 1245, the preference personalizationmodule 1250 may then generate personalized preferences in terms of routechoices. Such route related preferences may reflect not only routechoices but also represent preferences of route choices in differentsituations such as specific time frames of a day, seasons, locations,etc. Such established preferences for each individual driver may then bestored in storage 1265. At the time of route planning, the passengerpreference determiner 1240 receives the real-time data 480 and based onvarious types of information in the real-time data 480 (e.g.,month/day/time, passenger information, locale weather, etc.), thepassenger preference determiner 1240 may access, from the routeselection preference storage 1265, relevant preferences that can beapplied in current route planning. For example, if the real-time dataindicate that the driver is a specific person and the time is currently7:45 pm in January, etc., the passenger preference determiner 1240 mayidentify personalized route preferences in 1265 related to the currentspecific driver which are associated with the specific time frame andseason of the year (e.g., a driver may prefer driving on highway inwinter season). The personalized route selection preferences soidentified may then be sent to the route selection engine 1230 so thatpersonalized preferences of the driver/passenger at the time of theroute planning can be considered in determining what route is to beselected.

As shown in FIG. 12, the route selection engine 1230 may also take thepreferences estimated by the route selection preference determiner 1030as input and use that in its route selection operation. In someembodiments, the route selection engine 1230 may rely on the preferencesfrom 1030 without considering the personalized preferences of a driver,i.e., it may rely on merely the preferences identified by the routeselection preference determiner 1030 in its route selection.

In selecting route appropriate for the current situation, the routeselection engine 1230 may also receiving self-aware capabilityparameters 510. In the illustrated embodiment, the self-aware capabilityparameter analyzer 1205 separates extrinsic capability parameters andintrinsic capability parameters and sends the extrinsic capabilityparameters to the route selection engine 1230 so that extrinsicconditions associated with the current situation the vehicle is in canbe considered in selecting a route. For example, the extrinsiccapability parameters may indicate that there is on-going constructionon Route 7, the route selection engine 1230 may consider that and avoidRoute 7. However, if the destination is currently set for a school onRoute 7 and the driver's habit is to pick up children from the schooleach day at the present time (e.g., 3:30 pm), the route selection engine1230 may elect to choose Route 7, given all things considered.

Similarly, intrinsic capability parameters may also be considered inselecting an appropriate route. In this illustrated embodiment, theintrinsic capability parameters are fed to the intrinsic capabilitybased filter generator 1210, which may create different filters 1215based on the intrinsic capability parameters so that such filters may beutilized by the route selection engine to filter out routes that are notappropriate given the intrinsic capability parameters. For example, ifthe intrinsic capability parameters indicate that the vehicle has nosnow tire, any routes that are steep and/or curvy may not be appropriateon a snowy day.

The route selection engine 1230 selects a route based on both thecurrent location of the vehicle, tracked by a current location updater1235, and a destination, tracked by a destination updater 1225.Depending on the situation, changed current location and destination maytrigger the route selection engine 1230 to activate the route selectionpreference determiner 1030 to re-evaluate the preferences in routeselection given the changed circumstance.

FIG. 13 is a flowchart of an exemplary process for the global routeplanner 1020, according to an embodiment of the present teaching. At1310, self-aware capability parameters are received. Intrinsiccapability parameters are used to generate, at 1320, intrinsiccapability based filter(s) so that certain routes can be filtered out asinappropriate given the intrinsic conditions of the vehicle. Extrinsiccapability parameters are extracted, at 1330, from the receivedself-aware capability parameters. At the same time, real-time data 480are continuously received at 1340 and recorded human driving data arereceived at 1350. Such data are then used to determine, at 1360,personalized route selection preferences relevant to the current driver,current situation, and current time. At 1370, self-aware capabilityparameters and/or the personalized preferences of the driver may then beused to select a route appropriate given all factors considered. At1380, the selected route is output.

The route planning according to the present teaching allows varioustypes of information, such as real-time data and self-aware capabilityparameters, to be taken into account in route planning so that theplanned routes are adaptive with respect to the vehicle condition at thetime (via intrinsic capability parameters), the dynamic environment thevehicle is in at the time (via real-time data as well as extrinsiccapability parameters), the passenger characteristics determined basedon, e.g., dynamically updated real-time data (see FIG. 4B), as well aspassenger personalized preferences. Similarly, such information may alsobe utilized in other types of planning operations so that the plannedvehicle activities are adapt to the real-time situation, personalizedbased on individual preferences, and allows the vehicle to behave morehuman driver like. Below, more details are provided, with respect toFIGS. 14-19 on personalized adaptive motion planning.

Human drivers control their vehicle motion in a manner that iscomfortable. In most situations, human drivers also pay attention to thefeedback or reaction of passengers who ride with them in the vehicle andrespond to the vehicle motion. For example, some human drivers mayprefer start and stop the vehicle smoothly. Some human drivers whousually start and stop the vehicle fairly abruptly may adjust theirdriving when they observe that passengers sitting in their vehiclerespond in a certain way. Such human behavior may play an important rolein autonomous vehicles. It is commonly recognized that driving behaviorchanges from person to person and how such behavior is to be adjusted inthe presence of others in the same vehicle may also differ from personto person.

Traditionally, autonomous vehicles may adopt planning models that aretrained to capture the characteristics of human driving behavior of thegeneral population. Such generalized models do not customize theplanning approach based on individual driver/passenger preferences orintents. The present teaching aims to provide personalized motionplanning based on knowledge of the driver/passenger as well as thedynamic observations of driver/passenger response to vehicle motions.

FIG. 14A depicts an exemplary high level system diagram of the motionplanning module 560, according to an embodiment of the present teaching.In this illustrated embodiment, the motion planning module 560 aims atpersonalized, human-like, and adaptive motion planning, i.e., motions ofthe vehicle are planned in accordance with, e.g., general and personallikings, which may include what is known to be the preferences of apassenger and what is the reaction or feedback of a of a passenger ofthe current motion of the vehicle. The motion planning module 560according to the present teaching may comprise a generic motion planner1450 and a passenger motion adapter 1460. The motion planning module 560may plan vehicle motion based on various considerations, includingreal-time situations the vehicle is in (e.g., on a curvy road, rainingday, dim lighting, etc.), vehicle conditions (via intrinsic capabilityparameters), and personal preferences of the passenger in the vehicle(known preferences or dynamically determined based on driver feedbackobserved). Given those considerations, vehicle motion may be plannedbased on motion planning models, which may be invoked in a mannersuitable for different scenarios. Motion planning models may includedifferent models appropriate for the given situation in hand.

FIG. 14B illustrates exemplary types of motion planning models,according to an embodiment of the present teaching. In the illustratedembodiment, motion planning models may include generic motion planningmodels (1450 in FIG. 14A), sub-category models (1480 in FIG. 14A), orpersonalized models (1430 in FIG. 14A). The generic motion planner 1450may be preference based models or impact based models (see FIG. 14B). Apreference based model may be provided specifying preferred vehiclemotion in different scenarios based on general knowledge about vehicleoperation. For instance, when roads are slippery or icy, it is preferredto plan motions that are slower without sharp turns. An impact basedmodel may specify which kind of motion may cause what type of impact andsuch specification may be used to guide motion planning to achieve oravoid certain impact.

A sub-category model for motion planning may be, as compared with thegeneric models, directed to a sub-category of vehicles or a sub-categoryof drivers/passengers. For example, a sub-category model may be directedto sports cars and another sub-category model may be provided for vans.In addition, a sub-category model may be directed to teenager driversand another sub-category model may be directed to senior citizens. Eachsub-category model is tuned or specialized so that the motion planningfor a matching sub-category can be performed more accurately. Accordingto the present teaching, motion planning models may also includepersonalized models which may include individual models, each of whichmay specify preferences of each individual in terms of vehicle motions.For example, a passenger's individual preference model may specify thatthe passenger prefers smooth vehicle motion and another passenger'sindividual preference model may specify some different preferences. Suchgeneric, sub-category and individual models for motion planning may bederived based on recorded human driving data so that the motion plannedbased on such models are more human-like.

With reference back to FIG. 14A, in operation, to achieve personalized,human-like, and adaptive motion planning, the motion planning module 560receives various types of information and utilizes different motionplanning models. The received information includes planned route 520from the route planning module, surround information 420, real-time data480, and self-aware capability parameters 510. Based on the real-timedata 480 and the self-aware capability parameters 510, the genericmotion planner 1450 determines the situation the vehicle is in (e.g.,raining, dark, etc.) and accordingly invokes appropriate generic motionplanning models in 1440 to retrieve general motion planning information.In some embodiment, the generic motion planner 1450 may also determinerelevant sub-category of the vehicle and/or passenger so that associatedsub-category motion planning models may be retrieved from 1480 andutilized for motion planning. The generic motion planning models 1440may specify general motion planning strategies, e.g., if it is a snowyday or the vehicle is on a curvy road, it is preferred to make thevehicle motion slower and steady. Each sub-category model may beprovided to specify the generic motion planning strategies for thesub-category (e.g., a type of cars such as sports car or a sub-group ofpassengers such as senior citizens).

The motion planned by the generic motion planner 1450 (based on genericmotion planning models and/or sub-category motion planning models) maybe further adjusted or adapted according to personalized preferences. Inthe illustrated embodiment, this is achieved by the passenger motionadapter 1460. There may be different ways to adapt the motion planned tomeet personalized preferences. In some embodiments, personalizedpreferences may be accessed from individual passenger models 1430. Ifthe identity of the passenger is known, the associated individualpassenger model for the passenger may be retrieved from 1430 and thespecified preferences in vehicle motion may be used to determine how toachieve personalized motion planning. For instance, an individual modelfor a particular passenger may indicate that the passenger prefers asmooth ride without taking risks.

Another way to achieve personalized motion planning is adaptivelyadjusting motion planning based on dynamically observed information. Asdiscussed previously with reference to FIG. 4B, real-time data 480includes information related to passenger characteristics, which can bepassenger condition, . . . , and/or passenger reaction cues. Passengercondition may refer to mental, physical, and functional state of thepassenger. Information to be used in personalized motion planning mayalso include other types of data collected related to the situation. Apassenger observation analyzer 1420 may collect various types ofinformation and extract relevant indications and then send suchindications to the passenger motion adapter 1460 so that such dynamicand personalized information can be taken into account in motionplanning. Details about the passenger observation analyzer 1420 areprovided with reference to FIGS. 15-16.

FIG. 14C illustrates exemplary types of observations collected forconsideration in motion planning, according to an embodiment of thepresent teaching. Observations may include explicit expression from,e.g., the passenger such as voice or text input which may explicitlyindicate what the passenger desires. For instance, the passenger mayshout “faster!” or “I am going to be late!” or “I really get there ontime.” Such explicit expression detected may be relied on in motionplanning. The observations may also include detected scenarios that mayindicate something in terms of motion planning. Scenario information mayinclude event involved (which may indicate the urgency of the passengeris faced with), the current state of the passenger (such as age, knownhealth conditions, etc.), time of day which may imply certain task thepassenger has at that time (e.g., pick up children from school) whichrequires certain safety level. Observations may also include observedphysical reactions of the passenger which may be considered as relevantfor motion planning. For instance, sensors inside the vehicle maycapture any data that may indicate the emotion of the passenger, thebody language of the passenger, or the tone in passenger's voice, all ofwhich may reflect the desire of the passenger in response to the currentvehicle motion. For instance, the passenger may appear to be uneasy oreven show anxiety, which may indicate that the vehicle motion is toorough for the passenger. A sharp tone in passenger's voice may indicatethe same. Certain physical behavior may also suggest certain reactionfrom the passenger to the vehicle motion. For example, if the passengeris napping, yawning, looking sleepy, or reading, it may indicate thatthe passenger is comfortable with the vehicle motion. On the other hand,if it is observed that the passenger keeps checking watch, it mayindicate that the passenger feels the vehicle is moving too slow.

According to the present teaching, in addition to personalized motionplanning (e.g., not only with respect to sub-categories but also withrespect to individuals), motion planning may also be adaptive to thecurrent situation characterized by, e.g., self-aware capabilityparameters and real-time situations such as weather, road conditions,etc. The passenger motion adapter 1460 receives the extrinsic capabilityparameters from 1410 and plans motion accordingly. For example, ifextrinsic capability parameters indicate that there is sun glare orfoggy, motion may be planned accordingly (e.g., slow down).

FIG. 15 depicts an exemplary high level system diagram of the passengerobservation analyzer 1420, according to an embodiment of the presentteaching. In this illustrated embodiment, the passenger observationanalyzer 1420 is provided for obtaining the dynamic preferences of thepassenger in terms of vehicle motion so that the motion planning can beadapted to the personal likings. The dynamic preferences of thepassenger are derived based on analysis of the passenger's reaction cuesto the current vehicle motion, observed via different sensors. Theexemplary passenger observation analyzer 1420 comprises a sensoractivator 1500, a plurality of in-situ sensors 1510, a passengerdetector 1520, a passenger feature detector 1540, a visual-basedreaction cue estimator 1580, an acoustic-based reaction cue estimator1590, a passenger expression detector 1560, a passenger scenariodetector 1570, and a user reaction generator 1595.

The passenger observation analyzer 1420 is provided to determine thereaction or feedback of the passenger to the current vehicle motion todetermine whether the vehicle motion needs to be adjusted. For example,if passenger reaction indicates that the passenger is not happy aboutthe current vehicle motion, an adjustment may be made in motion planningaccordingly. The passenger reaction is to be estimated based ondifferent cues, including visual, acoustic, text, or contextualscenarios.

In some embodiments, the sensor activator 1500 activates the in-situsensors 1510 to detect the passenger reaction. The in-situ sensors 1510comprise a plurality of sensors including visual sensors, acousticsensors, infrared sensors, . . . , or communication sensors and the likethat enable the detection of any expression of the passenger. Forinstance, the visual sensors included in the in-situ sensors 1510 maycomprise a plurality of spatially distributed (within the vehicle)camera devices that are capable of capturing, processing and fusingimages of a scene from a variety of viewpoints into some form moreuseful individual images/videos. For example, the visual sensors maycapture a gesture or facial expression of the passenger, which may beused to estimate the reaction of the passenger. The in-situ sensors maybe selectively activated. For instance, at night, in order to observeaccurately the passenger's reaction, visual sensor may not work well andin this situation, infrared sensors may be activated instead.

As depicted in FIG. 14C, various physical reactions may be observed andused to analyze passenger's reaction cues. The passenger detector 1520receives sensor data and detects the passenger based on, passengerdetection models 1530. The detection may be based on either visual oracoustic information. Thus, the passenger detection models 1530 mayinclude both visual and acoustic models associated with the passengerand can be either individually invoked to detect the passenger based ona single modal data or both invoked to detect the passenger based onboth visual and acoustic features. For instance, the passenger detectionmodels 1530 may include a face recognition model which can be used todetect the passenger based on video or pictorial data from one or morevisual sensors. The passenger detection models 1530 may also include aspeaker based passenger detection model by which the passenger may berecognized based on his/her voice.

Upon the detection of the passenger, the sensor data may be continuouslyfed to the passenger feature detector 1540 to detect various passengerbehavior features, which may include both visual and acoustic. Forinstance, certain body language may be detected that may reveal that thepassenger is doing certain things, such as sleeping (napping), reading,yawning, or frequently checking his/her watch. Such detected passengerfeatures may also include acoustic features. For instance, the passengerfeature detector 1540 may detect that the passenger is saying “slowdown.” Visual and acoustic cues may be simultaneously detected thatreveal consistent reaction cues. For example, the passenger mayconstantly check the watch and say “faster!”

The passenger features may be detected based on visual and acousticfeature detection models 1550. Such models may guide the passengerfeature detector 1540 in terms of what feature to detect and provide,for each feature to be detected, a corresponding model that can be usedto detect the feature. Those models may be personalized in the sensethat what is to be detected may depend on the passenger. For instance,if the passenger is known to be mute, there is no reason to detectacoustic features associated with the passenger. Those feature detectionmodels may be adaptive so that once they are trained and deployed on thevehicle, they may be configured to receive scheduled or dynamic updateso that the models are adaptive to the changing situations.

The detected passenger visual features are then sent to the visual-basedreaction cue estimator 1580, which may then estimate the passenger'sreaction cues based on such visual cues. For example, if it is detectedthat the passenger is checking on the watch, the visual based reactionestimator 1580 may the reaction cue may be that the passenger is nothappy with the speed of the vehicle and becomes impatient. Such anestimated cue may also be derived based on, e.g., a personalized visualfeature model in 1550, which may be used to determine whether such abehavior (checking watch) is indicative of certain reaction cueassociated with this particular passenger (may or may not be dependingon the person).

Similarly, the detected passenger acoustic features are sent to theacoustic based reaction cue estimator 1590, which may then estimate thepassenger's reaction cue based on such acoustic features. For example,if it is detected that the passenger is snoring, the acoustic basedreaction cue estimator 1590 may estimate that the passenger iscomfortable with or at least not unhappy with the current vehiclemotion. Such an estimated cue may also be derived based on, e.g., apersonalized acoustic feature model in 1550, which may be used todetermine whether such a behavior is indicative of certain reaction cueof this particular passenger.

To estimate the reaction of the passenger, the visual based and acousticbased reaction cue estimators 1580 and 1590 may be engaged in estimatingthe emotional state of the passenger. For example, from the bodylanguage observed from the passenger (e.g., restless or seems to bevomiting), it may indicate that the passenger feels uneasy which may bea cue of his/her reaction to the vehicle motion. In addition, the toneof the voice used when the passenger says “faster” or “slow down” mayalso be used to estimate a level of anxiety of the passenger which is aclue as to how unhappy the passenger is. Such estimated emotional statemay be used in evaluating the severity of the reaction that thepassenger exhibits in response to the current vehicle motion and may beused to guide whether and/or how to adjust the motion planning.

In addition to the observed physical features, other parameters may alsobe used to estimate whether the current vehicle motion is acceptable.For instance, the source of observation may be an input directly enteredby the passenger via some communication interface (e.g., a touch screendisplay) within the vehicle. The passenger may input via the displayinterface in the vehicle that he/she wants the vehicle motion to besmoother. This may be detected by the passenger expression detector 1560via different communication sensors, which could be textual or acoustic.

As discussed previously, the scenarios that the passenger is currentlyin may also influence how the motion should be planned. The passengerscenario detector 1570 is configured to detect any scenario parametersthat may be relevant to motion planning. For instance, if it is knownthat each afternoon between 3:30 pm and 4:30 pm (time of day), thevehicle is used to pick up children from school (task in hand), this mayplace a restriction on motion planning. That is, the motion planned mayneed to be based on safety. Once detected, such a restriction may beconfigured to trump the estimated desire (of the passenger) to be fasterin order to ensure safety of the children. Other scenario relatedfactors may also be observed, e.g., the health and age of the passenger.If it is observed (from a passenger model 1535) that the passenger is anelderly and suffers from dementia, such scenario parameters may be usedto void some detected desire of the passenger. For instance, if thecurrent vehicle motion is already pretty fast and the passenger keepsdemanding to be even faster, given the age and known health condition ofthe passenger, the motion planning module may use such information tomake an appropriate motion planning decision.

The various passenger reaction cues, detected by the visual/acousticbased reaction cue estimators 1580 and 1590, the passenger expressiondetector 1560, and the passenger scenario detector 1570, are then sendto the user reaction generator 1595, where the detected differentparameters are selected and integrated to generate estimated userreaction and sent to the passenger motion adapter 1460 so that themotion planned by the generic motion planner 1450 may be adapted inaccordance with the observed dynamic user reaction to the currentvehicle motion.

FIG. 16 is a flowchart of an exemplary process for the passengerobservation analyzer 1420, according to an embodiment of the presentteaching. To gather observations associated with the passenger,appropriate sensors are activated at 1610. Information from activatedsensors is processed at 1620. To ascertain the physical behavior of thepassenger, the passenger is detected, at 1630, based on passengerdetection models 1530. Once the identity of the passenger isascertained, different types of features associated with the passengermay be obtained. At 1640, any explicit expression from the passenger isdetected. The scenario parameters associated with the passenger aredetected at 1660. Such gathered explicit expression from and scenarioparameters related to the passengers are then sent to the user reactiongenerator 1595.

Visual/acoustic features of the passenger are detected at 1650 and areused to estimate, at 1670 and 1680 respectively, the visual and acousticreaction cues, which are then sent also to the passenger reactiongenerator 1595. Different types of information so collected (from 1640,1660, 1670, and 1680) are then all used by the passenger reactiongenerator 1595 to generate, at 1690, the estimated user reaction.

Going back to FIG. 14, the estimated user reaction output by thepassenger observation analyzer 1420 is sent to the passenger motionadapter 1460 so that the real-time passenger reaction to the currentvehicle motion may be considered in determining how to adapt the plannedmotion based on the dynamic feedback from the passenger.

FIG. 17 is a flowchart of an exemplary process for the motion planningmodule 560, according to an embodiment of the present teaching. At 1710and 1720, real-time data 480 and the self-aware capability parametersare received, respectively. The self-aware capability parameters areprocessed, at 1730, and split into intrinsic and extrinsic capabilityparameters. Based on real time data and the intrinsic capabilityparameters, the generic motion planner 1450 generates, at 1740, plannedmotion for the vehicle. Such planned motion is generated based ongeneric motion planning models 1440 as well as any applicablesub-category motion planning models 1480.

To personalize the motion planning, the generically planned motion maythen be adapted based on personalized information, which may includeboth known personal preferences and dynamically observed passengerreaction to the current vehicle motion. To achieve that, know passengerpreferences are identified, at 1750, based on the individual passengermodels 1430. In addition, dynamic passenger reaction/feedback isestimated, at 1760, based on information gathered from differentsources/sensors. The personal preferences, either already known orestimated dynamically, are then used to personalize the motion planned,e.g., by adapting, at 1770, the motion planned based on genericinformation. Such personalized planned motion is then output, at 1780,as the planned motion 530.

As discussed with respect to FIG. 14A, various models are used in motionplanning, some being generic, some being semi-generic (sub-categorymodels are semi-generic), and some being personalized. In addition tobeing personalized and adaptive, the motion planning scheme as disclosedherein also aims at behaving in a manner that is more human-like. Beingadaptive to the dynamic reaction of the passenger may be part of it. Insome embodiments, the models used by the motion planning module 560 mayalso be generated to capture human-like behavior so that when they areapplied in motion planning, the planned motion 530 will be morehuman-like.

FIG. 18 depicts an exemplary high level system diagram of a motionplanning model training mechanism 1800 for generating those models,according to an embodiment of the present teaching. In this illustratedembodiment, the motion planning model training mechanism (MPMTM) 1800comprises data pre-process portion and a model training portion. Thedata pre-processing portion comprises a sub-category training dataclassifier 1820, an individual training data extractor 1830, and anobservation segmenter 1850. The model training portion comprises a modeltraining engine 1810 and an independent impact model training engine1840, and an independent impact model training engine 1840.

Recorded human driving data 430 are utilized to train models so that themodels can capture characteristics related to motion planning that aremore human-like. To train the generic motion planning models 1440, thereceived recorded human driving data are sent to the model trainingengine 1810 and the trained models are saved as the generic motionplanning models 1440. To obtain sub-category motion planning models1480, the recorded human driving data 430 are classified by thesub-category training data segmenter 1820 into training data sets forthe sub-categories and then fed to the model training engine 1810 fortraining. For each sub-category model, appropriate sub-category trainingdata set is applied to derive the corresponding sub-category model andsuch trained sub-category models are then saved in 1480. Similarly, forobtaining individual passenger models 1430 for motion planning, recordedhuman driving data may be processed to generate different training setsby the individual training data extractor 1830, each for an individual,and used by the model training engine 1810 to derive individualpassenger models that characterize the preferences of the correspondingindividuals.

In addition to the individual preferences, the individual passengermodels 1430 may also include models that characterize impact of vehiclemotions on individual passengers observed from the reaction or feedbackof passengers. The observed reaction/feedback may be positive ornegative and can be used to influence how the motion should be plannedin the future for passengers. FIG. 19 illustrates different types ofreactions observed and their roles in model training, according to anembodiment of the present teaching. For example, passengerreaction/feedback that can be used to train impact based models mayinclude negative or position impact. Negative reaction (negativereinforcement) of a passenger to certain planned motion may be capturedin a model so that similar motion may be avoided in the future as tothis particular passenger. Similarly, positive reaction to a plannedmotion or positive reinforcement observed may also be captured in themodel for future motion planning. Some reaction may be neutral which mayalso be captured by the individual passenger models.

To obtain impact based models for individuals, the real-time data 480,which capture the passenger characteristics in terms of theirbehavioral, visual, acoustic cues as well as their conditions (includingmental, physical and functional states during vehicle movement), may besegmented based on individuals and such segmented data may then be usedto derive models that characterize how certain motions impactpassengers. In some embodiments, the mechanism 1800 includes anobservation classifier 1850 that segments the real-time data 480according to individual passengers and fed such segmented training datasets to the independent impact model training engine 1840 to deriveindividual impact models. Such derived individual impact models are thenstored as part of the individual passenger models 1430.

Referring back to FIG. 5, the planning module 440 also includes laneplanning module 570, which may be directed to lane following and lanechanging as illustrated in FIG. 20A. Lane following may refer to thebehavior to stay in the lane while the vehicle is moving. Lane changingmay refer to the behavior to move from one lane the vehicle is currentlyin to an adjacent lane while the vehicle is moving. Lane planning mayrefer to the planning of vehicle behavior in terms of either lanefollowing or lane changing.

FIG. 20B illustrates exemplary types of behavior related to lanefollowing, according to an embodiment of the present teaching. Asillustrated, there are multiple lanes (2010, 2020, and 2030) and avehicle in each lane may follow its lane. Depending on differentsituations, the lane following behavior of individual vehicles maydiffer. For instance, as shown, when a vehicle merely tries to stay in alane without turning, the vehicle may behave to stay in the middle ofthe lane. This is shown in FIG. 20B with respect to the vehicles in lane2010 and 2020. This may be referred to as normal behavior 2040. When avehicle in lane 2030 needs to turn to the right, e.g., as shown 2050 inFIG. 20B, the vehicle in lane 2030 may behave differently. For example,instead of remain in the middle of lane 2030, vehicle in that lane maypull to the right side of the lane before the turn so that the turn is,e.g., safer and easier. Similarly, when a vehicle is turning left, thelane following behavior may differ as well. The lane planning module 570is configured to capture, e.g., via modeling, lane following behavior indifferent situations so that the autonomous driving vehicle may becontrolled in a natural and human-like way.

On the other hand, lane changing may involve behavior of the vehiclewhen it moves from one lane to an adjacent lane while the vehicle ismoving. Different passengers may exhibit different lane changingbehaviors. From safety considerations, there may be desirable lanechanging behaviors for different situations. Lane planning in terms oflane changing is to plan the vehicle movement with respect to the lanesin a manner that is safe, natural, human-like, and personalized.

FIG. 20C illustrates exemplary types of behavior related to lanechanging, according to an embodiment of the present teaching.Illustrated are different lane changing behavior, i.e., changing from acurrent lane 2020 to the lane left to it (lane 2010) and changing fromthe current lane 2020 to the lane right to it (lane 2030). With respectto the lane changing from lane 2020 to lane 2010, different lanechanging behaviors may be characterized in terms of (1) how fast to makethe change and (2) in what manner the vehicle is to move to the nextlane. For instance, as shown in FIG. 20B, there are illustrated threeplans to move to the lane 2010, which are left lane changing behavior 12060, left lane changing behavior 2 2070, and left lane changingbehavior 3 2080, representing respectively different speeds to move tolane 2010. By behavior 2060, the vehicle is to move to lane 2010fastest. By behavior 2080, the vehicle is to move to lane 2010 slowest.The speed to move to lane 2010 by behavior 2070 is in the middle.Similarly, when a vehicle is to move from lane 2020 to the lane to itsright (2030), there may also be different lane changing behavior, e.g.,right lane changing behavior 1 2065, right lane changing behavior 22075, and right lane changing behavior 2085, as shown in FIG. 20B.

In addition to the speed by which the vehicle is to move to the nextlane, the lane changing behavior may also differ in terms of how thevehicle moves into the next lane. Also as shown in FIG. 20B, when thevehicle is to move from lane 2020 to lane 2010 by employing left lanechanging behavior 1 2060, there are different behaviors for the vehicleto adopt to ease into lane 2010, e.g., by following a straight line2061, by following curve 2062 (cut in first and then straight out thevehicle), or by following curve 2063 (ease towards the edge of lane 2020first and watch and then cut in when ready). So, with regard to lanechanging, decisions as to vehicle behavior may be made at differentlevels.

Different drivers/passengers may exhibit different lane planning(include both lane following and lane changing) behaviors and in somesituations, the same driver/passenger may behave differently underdifferent circumstances. For instance, if there is no one on the street,a driver may decide to cut into the next lane quickly in lane change.When the street is crowded, the same driver may be more careful anddecide to take time to gradually ease into the next lane. The laneplanning module 570 is configured to learn different human behaviors indifferent circumstances and use such learned knowledge/models to achievelane planning in autonomous driving.

Smooth and predictable lane following and lane changing behavior is akey aspect of offering human-like driving experience in autonomousvehicles. It may be especially challenging when significant environmentnoise is present in camera images and/or videos captured during thevehicle operation. Traditional approaches rely on computer vision todetect lanes by detecting drivable area on the fly. Some uses end to endimage raw pixels for vehicle control signal prediction. Suchconventional approaches fail to utilize the available manual drivingdata collected so that they usually produce rigid planning and controland are susceptible to environment variances, while ultimately limit thecapability to operate the vehicle satisfactorily.

The present teaching utilizes lane detection models and lane planningmodels for lane planning and control. Both models are trained based onlarge amount of training data, some labeled and some are as collected.For lane detection, lane detection models are obtained using trainingdata with labeled lanes to derive supervised models for lane detection.Such supervised models are to be trained using a large set of trainingdata covering a wide range of environmental conditions to ensure therepresentativeness and robustness of the trained models.

For lane planning, to achieve human-like lane planning behavior, largevolume of human driving data are collected and used to train lanecontrol models that, when used for lane planning, are to exhibithuman-like behavior in maneuvering the vehicles. Although the lanedetection models and the lane planning models are trained separately, inoperation, the two sets of models are used in a cascade manner forinference in order to produce robust behavior in diverse types ofenvironment or conditions with human-like operational behavior. In someembodiments, when human driving data are classified according toindividual drivers, the present teaching can be configured to furtherpersonalize to create personalized human-like lane planning models. Withsuch personalized human-like lane planning models, an autonomous drivingvehicle may be operated during lane planning/control in an adaptivemanner, depending on who is the passenger in the vehicle.

FIG. 21 depicts an exemplary high level system diagram of the laneplanning module 570, according to an embodiment of the present teaching.In this illustrated embodiment, the lane planning module 570 comprisestwo model training engines 2110 and 2140 for training lane detectionmodels 2120 and lane planning models 2150, respectively. Such trainedmodels are then used, in lane planning, in a cascaded manner by adriving lane detector 2130 and a driving lane planning unit 2160. Asdiscussed above, the lane detection models 2120 are supervised modelsand are trained using training data with labeled lanes. Such supervisedtraining data are processed and used by the driving lane detection modeltraining engine 2110 to obtain the driving lane detection models 2120.

In some embodiments, the lane detection models 2120 may correspond to ageneric model, capturing the characteristics of lane detection indifferent situations. In some embodiments, the lane detection model 2120may include different models, each of which may be for providing a modelto detect lanes in a specific distinct situation. For example, somemodel(s) may be for detecting lanes in normal road conditions, some maybe for detecting lanes when the road is wet, some may be for detectinglanes when the road has glare or reflection, some may even be forestimating lanes when the roads are covered with, e.g., snow or othertypes of visual obstructing objects. The lane detection models may alsoprovide separate models for different types of vehicle. For example,some vehicles have higher gravity so that cameras capturing the groundimage in front of the vehicle may be installed at higher positionsrelative to the ground. In this case, the lane detection models for suchvehicles may be different from the lane detection models for vehicleswith cameras installed at a level closer to the ground level. Each typeof model may be trained using appropriate labeled training data that arerelated to the corresponding scenario.

To achieve human-like lane planning behavior in autonomous driving, thedriving lane planning model training engine 2140 takes recorded humandriving data 430 as input and learns human-like behavior in terms oflane planning. As discussed above, such human driving data may becollected from a wide range of drivers/situations/conditions in orderfor the driving lane planning model training engine 2140 to learn andcapture the characteristics of a wide range of human driving behavior inlane planning/control. In some embodiments, the driving lane planningmodel training engine 2140 may optionally take some supervised trainingdata with labeled lanes as input, e.g., as seeds or some small set ofdata to drive the learning towards convergence more quickly.

Based on the recorded human driving data 430, the driving lane planningmodel training engine 2140 may learn and/or train models for both lanefollowing and lane changing. In some embodiments, for each of lanefollowing and lane changing, a generic model in 2150 for generic humanbehavior may be derived. In some embodiments, the lane planning modeltraining engine 2140 may also learn and/or train multiple models forlane planning, each of which may be for different known situations,e.g., lane following or lane changing for specific subgroups of thegeneral population, or for particular different driving environmentscenarios (wet road, dark light, crowded road). Such models forsubgroups of the general population may also be stored in 2150.

The human-like lane control models 2150 may also be personalized andstored in 2150. When multiple models are to be derived via training,lane human driving data that meet the condition associated with each ofdifferent model may be extracted and used to train the models. Forexample, lane planning (including lane following and lane changing)models for lane related behavior exhibited when driving on crowded roadsmay be learned based on human driving data related to lane drivingbehavior on crowded roads. The models for lane planning may also bepersonalized. To achieve personalization, the driving lane planningmodel training engine 2140 may derive a model for each individualpassenger (e.g., with respect to each of lane following and lanechanging) based on the passenger's past driving data. Optionally,information from a personal profile associated with the passenger mayalso be used during learning in order to obtain a model that is moreaccurately reflect the preferences of the passenger.

Such obtained different types of lane planning/control models may thenbe stored in the driving lane control model storage 2150. In someembodiments, different models for different situations may be organizedand indexed for easy identification and quick access in real time duringthe operation of the vehicle. In some embodiments, the driving lanedetection model training engine 2110 and the driving lane planning modeltraining engine 2140 may reside remotely from the vehicle and thelearning may be performed in a centralized manner, i.e., they may beoperating based on training data from different sources and the learningand update may be activated regularly. The trained models may be sent todistributed vehicles. In some embodiments, personalized models for laneplanning may be updated locally in each vehicle based on data acquiredlocally.

The training via both 2110 and 2140 engines may be achieved via anylearning mechanism including artificial neural networks, deep leaningnetworks, etc. Depending on the types and number of models to beobtained, each training engine may comprise a plurality of sub trainingengines, each for a specific (set of) models for some specific purposesand each may be configured and implemented differently in order toderiving the most effective models. Each training engine (2110 and 2140)may also include, in addition to learning, pre-processing mechanisms(not shown) for process the training data prior to being used bylearning mechanism to derive trained models. For example, it may includedata segmentation mechanism that segment the received training data intoseparate sets, each may be used for training a specific model directedfor a particular situation, e.g., the driving lane planning modeltraining engine 2140 may be configured to derive a generic model for thegeneral population, a personalized model for the driver/passenger of thevehicle, a model for lane planning in day light condition, a model forlane planning in night light condition, a model for lane planning in wetroad condition, and a model for lane planning for snowy day condition.In this case, the pre-processing mechanism may then first group thereceived recorded human driving data 430 into different groups, each ofwhich for one model planned so that the training engine may then use theappropriate training data group to learn the appropriate model. Themodels may be continuously updated when the new training data arrive.The update of the models may be performed by re-learning based on alldata received (batch mode) or by incremental mode.

Once the models, include the lane detection models 2120 and the drivinglane control models 2150, are generated, they are used to plan lanerelated behavior for an autonomous driving vehicle in a human-likemanner and in some instances personalized. As discussed previously, inoperation, the obtained driving lane detection models 2120 and thedriving lane control models 2150 are applied in a cascade manner. In theillustrated embodiments, when the vehicle is on the road, sensor(s)installed in the vehicle take pictures/videos of the road the vehicle iscurrently driving on and send such sensor data to the driving lanedetector 2130. In addition to the sensor data, the driving lane detector2130 may also receive the self-aware capability parameters 510. Via theself-aware capability parameters, the driving lane detector 2130 maydetermine various types of information, e.g., road condition, thevehicle's capabilities, etc., in order to determine how it may proceedin a way that is appropriate. For example, if it is night time of theday, which may be indicated in the extrinsic capability parameters, thedriving lane detector 2130 may proceed to invoke a lane detection modelthat is trained for detecting lanes in dark light situation to achievereliable performance.

Using the appropriately invoked lane detection model(s), the drivinglane detector 2130 estimates segments of the lanes from the sensor dataand optionally the estimated position of the vehicle. Such estimatedlane segments and the vehicle position are then sent to the driving laneplanning unit 2160, where appropriate driving lane planning model(s) maythen be applied in a cascade manner for planning the lane controlbehavior of the vehicle.

As discussed previously, lane planning includes both lane following andlane changing. In operation, lane planning is directed to eithercontrolling the vehicle behavior in lane following or the vehiclebehavior in lane changing. When the vehicle is in motion, the operationcontext may provide some indication as to whether lane following or lanechanging planning is needed. For instance, if the vehicle needs to exit,it may need first to get into an exit lane from a current lane that doesnot lead to the exit. In this case, lane changing is implied so that thetask involved in lane planning is for lane changing. In someembodiments, the passenger in the vehicle may also provide an explicitlane control decision to indicate lane changing, e.g., by turning on theturn signal. In some embodiments, an indication of lane changing mayalso be from the vehicle itself, e.g., the engine may experience someproblem so that the autonomous driving system may send a lane controldecision signal to the driving lane planning unit 2160, instructing toprepare for lane changing so that the vehicle can move to the emergencylane. In normal situations, the vehicle may assume a default mode oflane following in absence of any indication of entering into a lanechanging mode.

To perform lane planning, the driving lane planning unit 2160 receives,from different sources, various types of information (e.g., detectedlanes, estimated vehicle position, lane planning decision, andself-aware capability parameters 510) and proceeds to lane planningaccordingly. For example, if the lane control decision signal indicatesthat the current task is for lane following, models for lane followingare to be retrieved and used for planning. If the current task is forlane changing, then models for lane changing are to be used.

Similar to the driving lane detector 2130, the driving lane planningunit 2160 may invoke the generic lane planning model from 2150 for theplanning. It may also invoke different lane planning models that areappropriate for the situation in hand in order to enhance theperformance. As discussed earlier, the self-aware capability parameters510 provide both intrinsic and extrinsic capability parameters, whichmay indicate the weather condition, road condition, etc. which can beused by the driving lane planning unit 2160 to invoke appropriate laneplanning models for the planning. For example, if the current task isfor lane following with a right turn coming up, personalized human-likemodels for the passenger in the event of a right turn from the currentlane may be retrieved from 2150 and used to plan the vehicle behavior asto how to ease into a position in the current right lane and then make aright turn.

On the other hand, if the current task is for lane changing, the lanecontrol decision indicates that it is to change to the lane left to thecurrent one, and the self-aware capability parameters indicate heavyrain and flooded roads, then the driving lane planning unit 2160 mayappropriately access lane planning models trained for planning lanechanging behavior on very wet roads. In some embodiments, such tasks mayalso be carried out using generic lane changing models. Based onselected models for the tasks in hand, the driving lane planning unit2160 generates the planned lane control, which may then be sent to thevehicle control module 450 (FIG. 4A) so that the planned lane controlbehavior can be implemented.

The driving lane planning unit 2160 may also perform personalized laneplanning. In some embodiments, the passenger currently present in thevehicle may be known, e.g., either via driver/passenger information sentto the driving lane planning unit 2160 or via detection of the passenger(now shown) from the sensor data. Upon receiving such information aboutthe passenger, the driving lane planning unit 2160 may appropriatelyinvoke lane control models suitable for the passenger. Such invokedcustomized models may be a model for a subgroup that the passengerbelongs to or may be a model that is personalized for the passenger.Such customized models may then be used to control how the lane planningis performed in a personalized manner.

FIG. 22 is a flowchart of an exemplary process for the lane planningmodule 570, according to an embodiment of the present teaching. Toobtained lane detection models, training data with labeled lanes arefirst received at 2210 and such received supervised data are then usedto obtain, at 2230 via training, driving lane detection models. On theother hand, to obtain driving lane planning models, recorded humandriving data, and optionally together with personalized profileinformation, are received at 2220 and used to derive, at 2240 viatraining, lane planning models. Once the models are obtained, they maybe dynamically updated based on newly arrived training data (now shown).

During operation, when a vehicle is in motion, the sensors on thevehicle acquire sensor data including imagery of the road ahead of thevehicle with lanes present. Such sensor data are received at 2250 andare used to detect, at 2260, lanes in front of the vehicle based on thelane detection models. The relative position of the vehicle may also beoptionally estimated. Such detected lanes and optionally estimatedvehicle position may then be sent to the driving lane planning unit2160. At the driving lane planning unit 2160, various types ofinformation received at 2270, which include lane control decision,detected lanes, and self-aware capability parameters. Such informationis used to determine the lane planning models to be used so that thelane planning can be achieved, at 2280, based on appropriated selectedlane planning models.

By learning from human driving data, the learned lane planning modelscapture the characteristics of human behavior in lane planning so thatwhen such models are used in autonomous driving, the vehicle can becontrolled in a human-like manner. In addition, by further personalizinglane planning models based on relevant driving data of thepassengers/drivers, the lane planning behavior of the vehicle can becontrolled in a manner that is familiar and comfortable for thepassenger/driver in the vehicle.

With reference to FIGS. 5-22, details of the planning module 440 onroute planning, motion planning, and lane planning are disclosed. Theoutput of the planning module 440 includes the planned route 520 fromthe route planning module 550, the planned motion 530 from the motionplanning module 560, and the planned lane control 540 from the laneplanning module 570 (see FIG. 5). Such output may be sent to differentparts of the autonomous driving vehicle in order to carry out theplanned vehicle behavior. For instance, the planned route 520 may besent to the part of the vehicle that is responsible for guide thevehicle in terms of route control, e.g., such as the built in GPS. Theplanned motion 530 and the planned lane control 540 may be sent to thevehicle control module 450 (in FIG. 4) so that the planned vehiclebehavior as to motion and lane control may be carried out on the vehiclevia the vehicle control module 450.

When the motion and lane control planned to achieve human-like behavior,the vehicle control module 450 aims at delivering the planned action.According to the present teaching, the vehicle control module 450 alsoaims at learning how to control the vehicle according to the knowledgein terms of how the vehicle behaves or responds to different controlsignals in different situations so that the vehicle can be controlled toachieve the desired effect, including the planned vehicle behavior.Traditional approaches apply machine learning based control and derivevehicle dynamics models from classical mechanics, which often fail tomodel a variety of situations that occurred in real world. As aconsequence, it often leads to poor performance and in some situations,may cause dangerous consequences. Although some conventional approachesis designed to learn the vehicle dynamics models from historic data via,e.g., neural networks, are able to learn the vehicle dynamics models incommon scenarios, in some situations, such systems have made predictionsthat have substantial and unpredicted errors, which in real life can befatal.

The present teaching discloses an approach that enables both achievingaccurate simulation and safety of the vehicle performance. Instead ofdirectly learn the vehicle dynamics model from the historic data,classical mechanics model is used as backbone model and learn how toadjust the predicted result from the historic data. In addition,limitation to the adjustment to be made is specified as a way to preventa prediction result that significantly deviates from the normalsituations.

FIG. 23A depicts a system diagram of traditional approach for generatingvehicle control signal. To determine vehicle control signals needed forcontrol the vehicle to achieve certain target motion, a vehiclekinematic model 2310 is provided and used by a vehicle kinematic model(VKM) vehicle control signal generator 2320 based on the target motionand information about the current vehicle state. For example, if thecurrent vehicle state is 30 miles per hour and the target motion isreaching 40 miles per hour in the next 5 seconds, the VKM vehiclecontrol signal generator 2320 uses such information to determine, basedon the vehicle kinematic model 2310, what kind of vehicle control is tobe applied so that the velocity acceleration can enable the vehicle toachieve the target motion. The approach as depicted in FIG. 23A is basedon the traditional vehicle kinematic models 2310, which is merely amechanical dynamic model.

FIG. 23B depicts a high level system diagram of the vehicle controlmodule 450 in FIG. 4A, according to an embodiment of the presentteaching. The vehicle control module 450 as disclosed herein aims atproviding the ability of generating vehicle control signals that canenable human-like driving behavior for autonomous vehicles. As shown,the vehicle control module 450 comprises a human-like vehicle controlunit 2340 and a human-like vehicle control (HLVC) model 2330. To achievehuman like autonomous driving, the human-like vehicle control unit 2340receives recorded human driving data 430 to uses that for learning humanlike vehicle control and generate the HLVC model 2330 that characterizesvehicle control behavior that is humanlike.

With the HLVC model 2330 created, when the human-like vehicle controlunit 2340 receives information related to a target motion and thecurrent vehicle state, it generates a human-like vehicle control signalbased on the HLVC model 2330 with respect to the real time situationassociated with the vehicle (characterized by the real time data 480).When additional recorded human driving data 430 are made available, theHLVC model 2330 may be dynamically updated or re-trained so that itcaptures the characteristics of human vehicle control behavior in avariety of situations. The dynamic update of the HLVC model 2330 may betriggered via a model update signal as shown in FIG. 23B. The modelupdate signal may be triggered manually or automatically when certainconditions are met, e.g., set up with regular update with apre-determined internal or when additional data available for updateamounts to a certain level.

In some embodiments, the HLVC model 2330 may also be personalized. Thisis illustrated in FIG. 23C. The HLVC model 2330 in this case maycomprise a plurality of HLVC sub-models (e.g., 2330-1, 2330-2, . . . ,2330-N), each of which may correspond to a sub-population with, e.g.,similar characteristics, and trained based on a part of the recordedhuman driving data 430 that are related to the sub-population. Forinstance, a HLVC sub-model may be directed to sub-population for peoplewho may prefer to drive cautiously so that the model may be derivedbased on training data in the recorded human driving data 430 from thedriving record of a corresponding sub-population that exhibit cautiousdriving record. A HLVC sub-model may also be personalized (e.g., asub-model is for an individual) if appropriate training data are appliedto derive a personalized sub-model.

Details on the human-like vehicle control unit 2340 is disclosed belowwith reference to FIGS. 24-29. FIG. 24 depicts an exemplary internalhigh level architecture of the human-like vehicle control unit 2340,which comprises a human-like vehicle control model generator 2410 and ahuman like vehicle control signal generator 2420. The human like vehiclecontrol model generator 2410 takes recorded human driving data 430 asinput and uses that information for learning and training the HLVC model2330. Exemplary types of data extracted from the recorded human drivingdata 430 for training may include, e.g., the vehicle control dataapplied to the vehicle and the vehicle states, which may include boththe vehicles states prior to and after the vehicle control data areapplied.

Data to be used for deriving the HLVC model 2330 may also includeenvironment data that characterize the surrounding condition under whichthe vehicle control data yielded the corresponding vehicle state. Theenvironment data may include various types of information, e.g., roadcondition, whether condition, vehicle type and condition. In someembodiments, the environment data may also include information about thepassenger in the vehicle as well as characteristics of the passenger,e.g., gender, age, health situation, preferences, etc. All thesedifferent types of information from the human driving data may presentsome variables that may impact the passenger's vehicle control behavior.For instance, when the road is wet or slippery, human drivers mayexhibit different vehicle control behavior in terms of break the vehicle(e.g., apply pressure on the brake more slowly) than that when the roadis not slippery.

When the HLVC model 2330 is generated, it can be used by the human-likevehicle control signal generator 2420 to generate a vehicle controlsignal, when it receives a desired target motion, to yield human-likevehicle control behavior in achieving the desired target motion. Togenerate a human-like vehicle control signal, the vehicle control signalgenerator 2420 obtains real time data 480 comprising information aboutthe surrounding of the vehicle at the time of the desired target motionand use such information in invoking the HLVC model 2330 to generate ahuman-like vehicle control signal. As illustrated in the example above,a target motion may be to accelerate, in 5 seconds, the vehicle to 40miles per hour starting from the current velocity of 30 miles per hour.The real time data for that moment may indicate that the road thevehicle is on has a deep slope and the road is slippery because it iscurrently raining. Such real time data is relevant and may be providedas environment data to the HLVC model 2330. The human-like vehiclecontrol signal generator 2420 may invoke the HLVC model 2330 with suchparameters in order to obtain an inferred human-like vehicle controlsignal that enables the autonomous vehicle to achieve the desired targetmotion in a manner similar to human driving.

FIG. 25 is a flowchart of an exemplary process of the human-like vehiclecontrol unit 2340, according to an embodiment of the present teaching.To generate the HLVC model 2330, recorded human driving data arereceived at 2510 to obtain training data, which are used in a trainingprocess, at 2520, to derive, at 2530, the HLVC model 2330. Suchgenerated HLVC model 2330 is then used in operation when a requestedtarget motion is received, at 2550. Based on the requested targetmotion, the human-like vehicle control signal generator 2420 obtains, at2560, the real time data 480 in order to extract information related tothe vehicle at the time, including environment data, current vehiclestate, etc. In some embodiments, information related to the passenger(e.g., identification of the passenger or characteristics of thepassenger) in the vehicle may also be obtained (not shown) so that suchinformation may be used to personalize the process of generating anappropriate human-like vehicle control signal.

The information obtained by the human-like vehicle control signalgenerator 2420 may then be applied to the HLVC model 2330 to generate,at 2570, the human-like vehicle control signal in accordance with theHLVC model 2330. In the event of personalization, one or more specificHLVC sub-models appropriate for the situation may be invoked and used togenerate personalized human-like vehicle control signal. During theoperation, the human-like vehicle control unit 2340 may check, at 2540,whether an update triggering signal is present. If an update signal isreceived, determined at 2540, the human-like vehicle control modelgenerator 2410 proceeds to step 2510 to gather training data andre-train or adaptively adjust the HLVC model 2330 based on thedynamically collected human driving data.

FIG. 26 depicts an exemplary high level system diagram of the human-likevehicle control model generator 2410, according to an embodiment of thepresent teaching. In the illustrated embodiment, the human-like vehiclecontrol model generator 2410 comprises a training data processing unit2610, a VKM vehicle control prediction engine 2630, and a vehiclecontrol model learning engine 2640. The training data processing unit2610 takes the recorded human driving data 430 as input, processes theinput to generate training data 2620 to be used to train the HLVC model2330. The recorded human driving data 430 may include environment data2620-1, vehicle state data 2620-2, . . . , and current vehicle controldata 2620-3. The environment data 2620-1 may include information such asthe road condition such as the slope of the road, angle of the turn,slippage of the surface condition, wetness of the road, etc. Theenvironment data may also include information related to limitations tothe vehicle such as speed limit, time of the day, season, location, etc.Such environment data may be used as contextual information in trainingso that the HLVC model 2330 may learn human like vehicle controlbehavior in different context conditions such that the trained modelcharacterizes human-like vehicle control behaviors exhibited indifferent situations. If personalization is needed, the obtainedtraining data may also be classified into different sub-groups oftraining data (not shown), each of which may be used to train an HLVCsub-model specific for passengers that belong to that group.

The vehicle state data 2620-2 may include information characterizing thestate of the vehicle, including, e.g., position of the vehicle, velocityof the vehicle, roll/pitch/yaw of the vehicle, and steering angle of thevehicle, etc. The vehicle control data 2620-3 may provide informationcharacterizing the control applied to the vehicle, such as brake appliedwith a certain force, steering by turning the steering wheel by acertain angle, or throttle.

According to the present teaching, rather than training the HLVC model2330 to generate the vehicle control signal directly, the presentteaching combines or fuses the traditional kinematic model basedprediction approach with learning model, created by learning from humandriving data, as to how to adjust a vehicle control signal predictedusing the traditional kinematic model so that the adjustment yieldshuman-like vehicle control behavior. Such an integrated approach enablesnot only more accurate vehicle control but also human-like feel as tocontrol of the vehicle.

In learning the HLVC model 2330, the vehicle state data 2620-2 andvehicle control data 2620-3 are provided to the VKM vehicle controlprediction engine 2630 to predict the motion achieved because of thecontrol exercised. The VKM vehicle control prediction engine 2630performs the prediction based on based on the vehicle kinematic model2310, e.g., via traditional mechanical dynamics approach to generate VKMbased prediction signal, as shown in FIG. 26. The VKM based predictionis then sent to the vehicle control model learning engine 2640 to becombined with other information from the training data 2620 to learn.Based on the outputs of the VKM vehicle control prediction engine 2630and the training data 2620, the vehicle control model learning engine2640 trains the HLVC model 2330 in, e.g., in an iterative process untilthe HLVC model 2330 converges. Once the model 2330 converges, it is usedto derive human-like vehicle control signals given desired targetmotions.

As shown, the vehicle control model learning engine 2640 may betriggered by the model update signal. When it is activated, the vehiclecontrol model learning engine 2640 invokes the training data processingunit 2610 and the VKM vehicle control prediction engine 2630 to initiatethe training process. In some embodiment, any subsequent training basedon additional human driving data may be performed in a derivative manneror in a batch mode, i.e., re-train the HLVC model 2330.

FIG. 27 is a flowchart of an exemplary process for the human-likevehicle control model generator 2410, according to an embodiment of thepresent teaching. Recorded human driving data 430 are first received at2710. The received human driving data are processed, at 2720, to obtaintraining data. Some of the training data are used, at 2730, to generateVKM based prediction based on the traditional vehicle kinematic model2310. For the learning and fusion based training, various aspects of thetraining data are identified, at 2740, and used, at 2750, to train theHLVC model 2330. After convergence, the HLVC model 2330 is created at2760.

FIG. 28 depicts an exemplary high level system diagram of the human-likevehicle control signal generator 2420, according to an embodiment of thepresent teaching. As discussed herein, the human-like vehicle controlsignal generator 2420 aims to generate, for a specified target motion, ahuman-like vehicle control signal based on the HLVC model 2330 so thatwhen the human-like vehicle control signal is used to control thevehicle, the vehicle exhibits human-like vehicle control behavior. Inthis embodiment, the human-like vehicle control signal generator 2420comprises a VKM vehicle control signal inference engine 2810, a contextdata determiner 2820, and an HLVC model based fusion unit 2830.

In operation, upon receiving a target motion, the human-like vehiclecontrol signal, the VKM vehicle control signal inference engine 2810obtains the current state of the vehicle and generates a VKM basedvehicle control signal based on the vehicle kinematic model 2310. Asdiscussed herein, the use of the traditional approach to generate aninferred vehicle control signal based merely on the vehicle kinematicmodel 2310 aims at providing initially an inferred vehicle controlsignal based on purely on mechanical dynamics. To achieve human-likebehavior in vehicle control to achieve the target motion, the inferredVKM based vehicle control signal is to be further used as an input tothe HLVC model based fusion unit 2830, where the VKM based vehiclecontrol signal is used as the initial inferred result to be fused withthe HLVC based approach so that the VKM based vehicle control signal maybe adjusted in accordance with the learned HLVC model 2330.

The HLVC model based fusion unit 2830 may activate, upon receiving thetarget motion, the context data determiner 2820 to obtain anyinformation related to the surrounding of the vehicle. The context datadeterminer 2820 receives the real time data 480 and extracts relevantinformation such as environment data or passenger data, etc. and sendsto the HLVC model based fusion unit 2830. Based on the target motion,the current vehicle state, context information surrounding the vehicle,and the VKM based vehicle control signal inferred using the traditionalvehicle kinematic model 2310, the HLVC model based fusion unit 2830accesses the HLVC model 2330 based on such input data to obtain a fusedhuman-like vehicle control signal.

As discussed herein, the HLVC model 2330 may be created by learning thediscrepancies between VKM model based predictions and the observedinformation from the recorded human driving data 430. As such, what theHLVC model 2330 captures and learns may correspond to adjustments to bemade to the VKM based vehicle control signals to achieve human-likebehavior. As discussed previously, as learning process may createoverfitting situation, especially when the training data includeoutliers, to minimize the risks in vehicle control due to adjustment tothe VKM based vehicle control signal, the human-like vehicle controlsignal generator 2420 may also optionally include preventative measuresby limiting the adjustments to VKM vehicle control signals based on somefusion constraints 2840, as shown in FIG. 28. In this manner, thehuman-like vehicle control signal generated as a modified VKM basedvehicle control signal can maximize the likelihood as to human behavioryet minimize the risks in vehicle control.

In some embodiments, information about the passenger in the vehicle mayalso be extracted from real time data 480 and can be used to accesspersonalized HLVC sub-model related to the passenger, which may be aHLVC sub-model for a group that the passenger belongs or a completelypersonalized HLVC sub-model for the passenger). Using such apersonalized HLVC sub-model may allow the human-like vehicle controlsignal generator 2420 to generate personalized human-like vehiclecontrol signal so that the vehicle control carried out based on it canbe not only human-like but also to the personal liking of the passenger.

FIG. 29 is a flowchart of an exemplary process of the human-like vehiclecontrol signal generator 2420, according to an embodiment of the presentteaching. Target motion information and the vehicle state data are firstreceived at 2910. Based on the target motion and the vehicle state, thevehicle kinematic model 2310 is accessed, at 2920, and used to infer, at2930, the VKM vehicle control signal. Such inferred control signal basedon mechanical dynamic model is sent to the HLVC model based fusion unit2830. To yield a fused human-like vehicle control signal, the contextdata determiner 2820 receives, at 2940, real time data 480 and extracts,at 2950, relevant information related to the vehicle. Using the contextinformation as well as the VKM vehicle control signal, the HLVC modelbased fusion unit 2830 infers, at 2960, the human-like vehicle controlsignal based on the HLVC model 2330. Such inferred human-like vehiclecontrol signal is then output, at 2970, so that human-like vehiclecontrol to achieve the target motion may be carried out in a human-likemanner.

FIG. 30 depicts the architecture of a mobile device which can be used torealize a specialized system implementing the present teaching. Thismobile device 3000 includes, but is not limited to, a smart phone, atablet, a music player, a handled gaming console, a global positioningsystem (GPS) receiver, and a wearable computing device (e.g.,eyeglasses, wrist watch, etc.), or in any other form factor. The mobiledevice 3000 in this example includes one or more central processingunits (CPUs) 3040, one or more graphic processing units (GPUs) 3030, amemory 3060, a communication platform 3010, such as a wirelesscommunication module, storage 3090, one or more input/output (I/O)devices 3050, a display or a projection 3020-a for visual basedpresentation, and one or more multi-modal interface channels 3020-b. Themulti-modal channels may include acoustic channel or other mediachannels for signaling or communication. Any other suitable component,including but not limited to a system bus or a controller (not shown),may also be included in the mobile device 3000. As shown in FIG. 30, amobile operating system 3070, e.g., iOS, Android, Windows Phone, etc.,and one or more applications 3080 may be loaded into the memory 3060from the storage 3090 in order to be executed by the CPU 3040.

To implement various modules, units, and their functionalities describedin the present disclosure, computer hardware platforms may be used asthe hardware platform(s) for one or more of the elements describedherein. The hardware elements, operating systems and programminglanguages of such computers are conventional in nature, and it ispresumed that those skilled in the art are adequately familiar therewithto adapt those technologies to the present teachings as describedherein. A computer with user interface elements may be used to implementa personal computer (PC) or other type of work station or terminaldevice, although a computer may also act as a server if appropriatelyprogrammed. It is believed that those skilled in the art are familiarwith the structure, programming and general operation of such computerequipment and as a result the drawings should be self-explanatory.

FIG. 31 depicts the architecture of a computing device which can be usedto realize a specialized system implementing the present teaching. Sucha specialized system incorporating the present teaching has a functionalblock diagram illustration of a hardware platform which includes userinterface elements. The computer may be a general purpose computer or aspecial purpose computer. Both can be used to implement a specializedsystem for the present teaching. This computer 3100 may be used toimplement any component or aspect of the present teachings, as describedherein. Although only one such computer is shown, for convenience, thecomputer functions relating to the present teachings as described hereinmay be implemented in a distributed fashion on a number of similarplatforms, to distribute the processing load.

The computer 3100, for example, includes COM ports 3150 connected to andfrom a network connected thereto to facilitate data communications. Thecomputer 3100 also includes a central processing unit (CPU) 3120, in theform of one or more processors, for executing program instructions. Theexemplary computer platform includes an internal communication bus 3110,program storage and data storage of different forms, e.g., disk 3170,read only memory (ROM) 3130, or random access memory (RAM) 3140, forvarious data files to be processed and/or communicated by the computer,as well as possibly program instructions to be executed by the CPU. Thecomputer 2600 also includes an I/O component 3160, supportinginput/output flows between the computer and other components thereinsuch as interface elements 3180 in different media forms. An exemplarytype of interface element may correspond to different types of sensors3180-a deployed on the autonomous driving vehicle. Another type ofinterface element may correspond to a display or a projection 3180-b forvisual based communication. There may be additional components for othermulti-modal interface channels such as acoustic device 3180-c for audiobased communications and/or component 2680-d for signaling based oncommunication, e.g., signal that causes vibration on a vehicle componentsuch as a car seat. The computer 3100 may also receive programming anddata via network communications.

Hence, aspects of the methods of the present teachings, as outlinedabove, may be embodied in programming. Program aspects of the technologymay be thought of as “products” or “articles of manufacture” typicallyin the form of executable code and/or associated data that is carried onor embodied in a type of machine readable medium. Tangiblenon-transitory “storage” type media include any or all of the memory orother storage for the computers, processors or the like, or associatedmodules thereof, such as various semiconductor memories, tape drives,disk drives and the like, which may provide storage at any time for thesoftware programming.

All or portions of the software may at times be communicated through anetwork such as the Internet or various other telecommunicationnetworks. Such communications, for example, may enable loading of thesoftware from one computer or processor into another, for example, froma management server or host computer of a search engine operator orother enhanced ad server into the hardware platform(s) of a computingenvironment or other system implementing a computing environment orsimilar functionalities in connection with the present teachings. Thus,another type of media that may bear the software elements includesoptical, electrical and electromagnetic waves, such as used acrossphysical interfaces between local devices, through wired and opticallandline networks and over various air-links. The physical elements thatcarry such waves, such as wired or wireless links, optical links or thelike, also may be considered as media bearing the software. As usedherein, unless restricted to tangible “storage” media, terms such ascomputer or machine “readable medium” refer to any medium thatparticipates in providing instructions to a processor for execution.

Hence, a machine-readable medium may take many forms, including but notlimited to, a tangible storage medium, a carrier wave medium or physicaltransmission medium. Non-volatile storage media include, for example,optical or magnetic disks, such as any of the storage devices in anycomputer(s) or the like, which may be used to implement the system orany of its components as shown in the drawings. Volatile storage mediainclude dynamic memory, such as a main memory of such a computerplatform. Tangible transmission media include coaxial cables; copperwire and fiber optics, including the wires that form a bus within acomputer system. Carrier-wave transmission media may take the form ofelectric or electromagnetic signals, or acoustic or light waves such asthose generated during radio frequency (RF) and infrared (IR) datacommunications. Common forms of computer-readable media thereforeinclude for example: a floppy disk, a flexible disk, hard disk, magnetictape, any other magnetic medium, a CD-ROM, DVD or DVD-ROM, any otheroptical medium, punch cards paper tape, any other physical storagemedium with patterns of holes, a RAM, a PROM and EPROM, a FLASH-EPROM,any other memory chip or cartridge, a carrier wave transporting data orinstructions, cables or links transporting such a carrier wave, or anyother medium from which a computer may read programming code and/ordata. Many of these forms of computer readable media may be involved incarrying one or more sequences of one or more instructions to a physicalprocessor for execution.

Those skilled in the art will recognize that the present teachings areamenable to a variety of modifications and/or enhancements. For example,although the implementation of various components described above may beembodied in a hardware device, it may also be implemented as a softwareonly solution—e.g., an installation on an existing server. In addition,the present teachings as disclosed herein may be implemented as afirmware, firmware/software combination, firmware/hardware combination,or a hardware/firmware/software combination.

While the foregoing has described what are considered to constitute thepresent teachings and/or other examples, it is understood that variousmodifications may be made thereto and that the subject matter disclosedherein may be implemented in various forms and examples, and that theteachings may be applied in numerous applications, only some of whichhave been described herein. It is intended by the following claims toclaim any and all applications, modifications and variations that fallwithin the true scope of the present teachings.

We claim:
 1. A method, comprising: receiving, via a processor, a firstset of images generated from real time visual sensor data and depictinga ground surface on which an autonomous vehicle is positioned;detecting, in real time and based on the first set of images, marks onthe ground surface representing a lane that the autonomous vehicle iscurrently occupying based on at least one lane detection model obtainedvia machine learning; receiving, via the processor, data representing apresence of a passenger within the autonomous vehicle; selecting apersonalized lane control model from a plurality of personalized lanecontrol models related to the passenger, each personalized lane controlmodel from the plurality of personalized lane control models beingtrained via machine learning based on recorded human driving datarelated to the passenger under different driving conditions; generatinga planned lane control for the autonomous vehicle based on the lane thatthe autonomous vehicle is currently occupying and self-aware capabilityparameters associated with the selected personalized lane control model,the generating the planned lane control including using the self-awarecapability parameters to predict an operational capability of theautonomous vehicle with respect to a current location of the autonomousvehicle; and sending a signal to cause implementation of the plannedlane control at the autonomous vehicle.
 2. The method of claim 1,wherein the planned lane control includes at least one of lane followingor lane changing.
 3. The method of claim 1, wherein the self-awarecapability parameters include intrinsic capability parameters andextrinsic capability parameters, the intrinsic capability parametersspecifying conditions internal to the autonomous vehicle that limit theoperational capability of the autonomous vehicle, the extrinsiccapability parameters specifying conditions external to the autonomousvehicle that impact the operational capability of the autonomousvehicle.
 4. The method of claim 1, wherein the self-aware capabilityparameters are dynamically updated to reflect a scenario the autonomousvehicle is currently associated with.
 5. The method of claim 1, whereinthe lane detection model is selected from a plurality of lane detectionmodels based on the self-aware capability parameters.
 6. The method ofclaim 1, wherein the data representing the presence of the passengerwithin the autonomous vehicle is obtained by: receiving a second set ofimages generated from the real time visual sensor data and depicting aninternal scene of the autonomous vehicle; analyzing the second set ofimages to detect the presence of the passenger; and generating the datarepresenting the presence of the passenger within the autonomousvehicle.
 7. A non-transitory, processor-readable medium storinginstructions to cause a processor to: receive a first set of imagesgenerated from real time visual sensor data and depicting a groundsurface on which an autonomous vehicle is positioned; detect, in realtime and based on the first set of images, marks on the ground surfacerepresenting a lane that the autonomous vehicle is currently occupyingbased on at least one lane detection model obtained via machinelearning; receive data representing a presence of a passenger within theautonomous vehicle; select a personalized lane control model from aplurality of personalized lane control models related to the passenger,wherein each personalized lane control model from the plurality ofpersonalized lane control models is trained via machine learning basedon recorded human driving data related to the passenger under differentdriving conditions; generate a planned lane control for the autonomousvehicle based on the lane that the autonomous vehicle is currentlyoccupying and self-aware capability parameters associated with theselected personalized lane control model, the generating the plannedlane control including using the self-aware capability parameters topredict an operational capability of the autonomous vehicle with respectto a current location of the autonomous vehicle; and send a signal tocause implementation of the planned lane control at the autonomousvehicle.
 8. The medium of claim 7, wherein the planned lane controlincludes at least one of lane following or lane changing.
 9. The mediumof claim 7, wherein the self-aware capability parameters includeintrinsic capability parameters and extrinsic capability parameters, theintrinsic capability parameters specifying conditions internal to theautonomous vehicle that limit the operational capability of theautonomous vehicle, the extrinsic capability parameters specifyingconditions external to the autonomous vehicle that impact theoperational capability of the autonomous vehicle.
 10. The medium ofclaim 7, wherein the self-aware capability parameters are dynamicallyupdated to reflect a scenario the autonomous vehicle is currentlyassociated with.
 11. The medium of claim 7, wherein the lane detectionmodel is selected from a plurality of lane detection models based on theself-aware capability parameters.
 12. The medium of claim 7, wherein thedata representing the presence of the passenger within the autonomousvehicle is obtained by: receiving a second set of images generated fromthe real time visual sensor data and depicting an internal scene of theautonomous vehicle; analyzing the second set of images to detect thepresence of the passenger; and generating the data representing thepresence of the passenger within the autonomous vehicle.
 13. A system,comprising: a driving lane detector implemented by a processor andconfigured to: receive a first set of images generated from real timevisual sensor data and depicting a ground surface on which an autonomousvehicle is positioned, detect, in real time and based on the first setof images, marks on the ground surface representing a lane that theautonomous vehicle is currently occupying based on at least one lanedetection model obtained via machine learning; and a driving laneplanning unit implemented by the processor and configured to receivedata representing a presence of a passenger within the autonomousvehicle, select a personalized lane control model from a plurality ofpersonalized lane control models related to the passenger, wherein eachpersonalized lane control model from the plurality of personalized lanecontrol models is trained via machine learning based on recorded humandriving data related to the passenger under different drivingconditions, generate a planned lane control for the autonomous vehiclebased on the lane that the autonomous vehicle is currently occupying andself-aware capability parameters associated with the selectedpersonalized lane control model, the generating the planned lane controlincluding using the self-aware capability parameters to predict anoperational capability of the autonomous vehicle with respect to acurrent location of the autonomous vehicle; and send a signal to causeimplementation of the planned lane control at the autonomous vehicle.14. The system of claim 13, wherein the planned lane includes at leastone of lane following or lane changing.
 15. The system of claim 13,wherein the self-aware capability parameters include intrinsiccapability parameters and extrinsic capability parameters, the intrinsiccapability parameters specifying conditions internal to the autonomousvehicle that limit the operational capability of the autonomous vehicle,the extrinsic capability parameters specifying conditions external tothe autonomous vehicle that impact the operational capability of theautonomous vehicle.
 16. The system of claim 13, wherein the self-awarecapability parameters are dynamically updated to reflect a scenario theautonomous vehicle is currently associated with.
 17. The system of claim13, wherein the lane detection model is selected from a plurality oflane detection models based on the self-aware capability parameters. 18.The system of claim 13, wherein the data representing the presence ofthe passenger within the autonomous vehicle is obtained by: receiving asecond set of images generated from the real time visual sensor data anddepicting an internal scene of the autonomous vehicle; analyzing thesecond set of images to detect the presence of the passenger; andgenerating the data representing the presence of the passenger withinthe autonomous vehicle.